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BEST  PRACTICES 

An  Integrated  Portfolio  Management 
Approach  to  Weapon  System 
Investments  Could  Improve  DOD’s 
Acquisition  Outcomes 


What  GAO  Found 

To  achieve  a  balanced  mix  of  executable  development  programs  and  ensure 
a  good  return  on  their  investments,  the  successful  commercial  companies 
GAO  reviewed  take  an  integrated,  portfolio  management  approach  to 
product  development.  Through  this  approach,  companies  assess  product 
investments  collectively  from  an  enterprise  level,  rather  than  as  independent 
and  unrelated  initiatives.  They  weigh  the  relative  costs,  benefits,  and  risks  of 
proposed  products  using  established  criteria  and  methods,  and  select  those 
products  that  can  exploit  promising  market  opportunities  within  resource 
constraints  and  move  the  company  toward  meeting  its  strategic  goals  and 
objectives.  Investment  decisions  are  frequently  revisited,  and  if  a  product 
falls  short  of  expectations,  companies  make  tough  go/no-go  decisions.  The 
companies  GAO  reviewed  have  found  that  effective  portfolio  management 
requires  a  governance  structure  with  committed  leadership,  clearly  aligned 
roles  and  responsibilities,  portfolio  managers  who  are  empowered  to  make 
investment  decisions,  and  accountability  at  all  levels  of  the  organization. 

In  contrast,  DOD  approves  proposed  programs  with  much  less  consideration 
of  its  overall  portfolio  and  commits  to  them  earlier  and  with  less  knowledge 
of  cost  and  feasibility.  Although  the  military  services  fight  together  on  the 
battlefield  as  a  joint  force,  they  identify  needs  and  allocate  resources 
separately,  using  fragmented  decision-making  processes  that  do  not  allow 
for  an  integrated,  portfolio  management  approach  like  that  used  by 
successful  commercial  companies.  Consequently,  DOD  has  less  assurance 
that  its  investment  decisions  address  the  right  mix  of  warfighting  needs,  and, 
as  seen  in  the  figure  below,  it  starts  more  programs  than  current  and  likely 
future  resources  can  support,  a  practice  that  has  created  a  fiscal  bow  wave. 
If  this  trend  goes  unchecked,  Congress  will  be  faced  with  a  difficult  choice: 
pull  dollars  from  other  high-priority  federal  programs  to  fund  DOD’s 
acquisitions  or  accept  gaps  in  warfighting  capabilities. 


What  GAO  Recommends 


GAO  is  making  several 
recommendations  for  DOD  to 
implement  an  integrated  portfolio 
management  approach  to  weapon 
system  investments.  DOD  stated 
that  it  is  undertaking  several  pilot 
efforts  to  improve  the  department’s 
approach  and  that  implementation 
of  any  new  business  rules  will  be 
contingent  upon  the  outcomes  of 
these  efforts. 


www.gao.gov/cgi-bin/getrpt7GAO-07-388. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


Figure:  Costs  Remaining  Versus  Annual  Appropriations  for  Major  Defense  Acquisitions 
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Why  GAO  Did  This  Study 

Over  the  next  several  years,  the 
Department  of  Defense  (DOD) 
plans  to  invest  $1.4  trillion  in  major 
weapons  programs.  While  DOD 
produces  superior  weapons,  GAO 
has  found  that  the  department  has 
failed  to  deliver  weapon  systems 
on  time,  within  budget,  and  with 
desired  capabilities.  While  recent 
changes  to  DOD’s  acquisition 
policy  held  the  potential  to  improve 
outcomes,  programs  continue  to 
experience  significant  cost  and 
schedule  overruns. 

GAO  was  asked  to  examine  how 
DOD’s  processes  for  determining 
needs  and  allocating  resources  can 
better  support  weapon  system 
program  stability.  Specifically, 

GAO  compared  DOD’s  processes 
for  investing  in  weapon  systems  to 
the  best  practices  that  successful 
commercial  companies  use  to 
achieve  a  balanced  mix  of  new 
products,  and  identified  areas 
where  DOD  can  do  better.  In 
conducting  its  work,  GAO 
identified  the  best  practices  of: 
Caterpillar,  Eli  Lilly,  IBM,  Motorola, 
and  Procter  and  Gamble. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


March  30,  2007 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

Although  the  Department  of  Defense  (DOD)  produces  the  best  weapons  in 
the  world,  it  has  not  been  able  to  deliver  planned  systems  on  time  and 
within  budget.  It  is  not  unusual  to  see  cost  increases  that  add  up  to  tens  or 
hundreds  of  millions  of  dollars,  schedule  delays  that  add  up  to  years,  and 
large  and  expensive  programs  being  scrapped  after  years  of  failing  to 
achieve  promised  capabilities.1  While  recent  changes  to  DOD’s  acquisition 
policy  held  the  potential  to  improve  such  outcomes,  programs  have 
continued  to  experience  significant  cost  and  schedule  overruns  and 
performance  shortfalls.2  Over  the  next  several  years,  DOD  plans  to  invest 
$1.4  trillion  in  major  weapon  system  programs — doubling  what  it  planned 
to  spend  on  such  programs  5  years  ago.  Continued  failure  to  deliver 
weapon  systems  on  time  and  within  budget  not  only  delays  providing 
critical  capabilities  to  the  warfighter,  but  results  in  less  funding  being 
available  for  other  DOD  and  federal  priorities. 

In  the  commercial  market,  effectively  developing  and  marketing  new 
products  is  fundamental  to  the  continued  growth  and  success  of 
companies.  Without  a  steady  stream  of  product  innovations  to  meet 
evolving  market  needs,  companies  are  likely  to  see  their  sales  and  profits 
fall.  At  the  same  time,  if  the  products  in  development  outstrip  their 
resources  or  do  not  meet  customer  needs,  companies  can  face  financial 
min.  Several  recent  studies  issued  by  leading  thinkers  in  the  area  of 
product  innovation  and  development  have  reported  that  leading 
commercial  companies  achieve  success  in  product  development  by 


^AO,  Defense  Acquisitions:  Actions  Needed  to  Get  Better  Results  on  Weapons  Systems 
Investments,  GAO-06-585T  (Washington,  D.C.:  Apr.  5,  2006). 

2GAO,  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13, 
2006). 
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following  a  disciplined  process  for  ensuring  they  have  the  right  mix  of  new 
products  that  meet  customer  needs  within  available  resources. 

In  fiscal  year  2006,  the  Senate  Armed  Services  Committee  raised  concerns 
that  DOD’s  poor  track  record  with  acquisition  programs  was  linked  not 
only  to  the  department’s  Defense  Acquisition  System  (DAS)  for  managing 
product  development,  but  also  to  the  department’s  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  for  identifying  the 
warfighters’  needs  and  the  Planning,  Programming,  Budgeting  and 
Execution  (PPBE)  process  for  allocating  resources.  Consequently,  the 
Committee  directed  GAO  to  examine  how  DOD’s  needs  identification  and 
resource  allocation  processes  can  better  support  program  stability  in 
major  weapon  systems  acquisition.3  This  report  (1)  identifies  best 
practices  of  successful  commercial  companies  for  ensuring  that  they 
pursue  the  right  mix  of  programs  to  meet  the  needs  of  their  customers 
within  resource  constraints  and  (2)  compares  DOD’s  enterprise-level 
processes  for  investing  in  weapon  systems  to  these  practices.4 

To  identify  best  practices  of  successful  companies,  we  reviewed  related 
professional  and  academic  publications,  and  interviewed  knowledgeable 
officials  from  five  successful  commercial  companies:  Caterpillar,  Eli  Lilly, 
IBM,  Motorola,  and  Procter  &  Gamble.  While  the  products  developed  by 
these  companies  range  from  heavy  construction  equipment  and  high-end 
electronics  to  pharmaceuticals  and  household  items,  each  of  the 
companies  manages  a  large  diversified  portfolio  of  products,  spends 
billions  of  dollars  annually  on  research  and  development,  and  has 
thousands  of  employees  worldwide.  To  examine  DOD’s  processes  for 
making  investment  decisions,  we  reviewed  related  legislation  and  DOD 
directives,  instructions,  and  guidance;  conducted  interviews  with  and 
received  briefings  from  relevant  Joint  Staff,  Office  of  the  Secretary  of 
Defense  (Comptroller,  Program  Analysis  and  Evaluation,  and  Acquisition, 
Technology  and  Logistics),  and  other  government  officials;  reviewed 
current  literature  assessing  DOD’s  decision-making  processes;  and 
analyzed  DOD  requirements  documents.  We  compared  DOD’s  enterprise- 
level  practices  to  commercial  best  practices  to  identify  potential  areas  for 


3S.Rep.No.  109-69  at  343-346  (2005). 

4The  Senate  mandate  specifically  asked  GAO  to  review  the  requirements  and  budgeting 
portions  of  DOD  weapon  system  acquisition.  Therefore,  we  do  not  focus  on  the  DAS  in  this 
report.  GAO  has,  however,  produced  a  body  of  best  practices  work  focusing  on  the  DAS; 
many  of  those  reports  are  listed  in  the  related  GAO  products  section  of  this  report. 
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improvement.  For  additional  details  on  how  we  performed  our  review,  see 
appendix  I.  Our  work  was  conducted  between  March  2006  and  February 
2007  in  accordance  with  generally  accepted  government  auditing 
standards. 


Results  in  Brief 


Successful  commercial  companies  use  an  integrated  portfolio 
management  approach  to  prioritize  market  needs  and  allocate  resources; 
thus,  they  avoid  pursuing  more  products  than  their  resources  can  support 
and  optimize  the  return  on  their  investment.  Through  portfolio 
management,  all  of  a  company’s  product  investments  are  addressed 
collectively  from  an  enterprise  level,  rather  than  as  independent  and 
unrelated  initiatives.  Potential  product  developments  are  identified  and 
assessed  through  a  systematic  and  disciplined  screening  process. 
Companies  weigh  the  relative  costs,  benefits,  and  risks  of  each  proposed 
product  using  established  criteria  and  methods  to  select  the  best  mix  of 
products  to  develop.  They  not  only  select  those  products  that  have  a 
sound  business  case  to  warrant  further  investment,  but  also  those  that 
help  the  company  balance  near-  and  future-term  market  opportunities, 
different  product  lines,  and  available  resources  against  the  demand  for 
product  investments.  Once  initial  investment  decisions  are  made,  they  are 
revisited  at  multiple  stages  throughout  product  development  in  a  gated 
review  and  assessment  process  to  ensure  products  are  still  of  high  value. 
If  not,  companies  make  tough  decisions  to  defer  or  terminate  investments 
and  rebalance  their  product  portfolios.  To  be  effective,  portfolio 
management  is  enabled  by  strong  governance  with  committed  leadership, 
clearly  aligned  organizational  roles  and  responsibilities,  empowered 
portfolio  managers  who  determine  the  best  way  to  invest  resources,  and 
accountability  at  all  levels  of  the  organization. 

Although  the  military  services  fight  together  on  the  battlefield  as  a  joint 
force,  they  do  not  identify  warfighting  needs  and  make  weapon  system 
investment  decisions  together  in  an  integrated  manner.  DOD  has  taken 
steps  to  identify  warfighting  needs  through  a  joint  requirements  process, 
but  its  service-centric  structure  and  fragmented  decision-making 
processes  do  not  allow  for  the  portfolio  management  approach  used  by 
successful  commercial  companies  to  make  investment  decisions  that 
benefit  the  organization  as  a  whole.  DOD  largely  continues  to  define 
warfighting  needs  and  make  investment  decisions  on  a  service-by-service 
basis,  an  approach  that  has  contributed  to  duplication  in  programs  and 
equipment  that  does  not  operate  effectively  together.  Also,  DOD  assesses 
warfighting  needs  and  their  funding  implications  under  separate  decision¬ 
making  processes,  impeding  its  ability  to  prioritize  warfighting  needs  so 
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that  it  pursues  not  only  the  ones  that  are  most  important  but  also  the  ones 
it  can  afford.  While  DOD’s  JCIDS  process  provides  a  framework  for 
reviewing  and  validating  the  initial  need  for  proposed  capabilities,  it  does 
not  focus  on  the  cost  and  feasibility  of  acquiring  the  capability  to  be 
developed  and  fielded.  Instead,  these  considerations  are  addressed 
through  separate  budgeting  and  acquisition  processes.  Moreover,  although 
DOD  policy  provides  for  a  series  of  early  reviews — focused  on  the  concept 
refinement  and  technology  development  phases  of  proposed  weapon 
system  programs — in  prior  work  we  found  that  the  reviews  are  often 
skipped  or  are  not  fully  implemented.  Consequently,  proposed  programs 
build  momentum  and  move  toward  starting  product  development  with 
little  if  any  early  department-level  assessment  of  the  costs  and  feasibility. 
Committing  to  programs  before  they  have  this  knowledge  contributes  to 
poor  cost,  schedule,  and  performance  outcomes  and  destabilizes 
acquisition  programs  as  the  department  attempts  to  pay  for  poorly 
performing  programs  by  taking  funds  from  others. 

The  department  has  begun  to  pilot-test  several  interrelated  initiatives 
intended  to  address  shortfalls  in  its  existing  approach  to  investment 
decisions.  These  initiatives  include  a  new  approach  to  an  early  decision 
gate  for  reviewing  proposed  programs  at  the  concept  stage,  testing 
portfolio  management  approaches  in  selected  capability  areas,  and  setting 
up  capital  budgeting  accounts  for  programs  in  development.  However,  as 
currently  structured,  the  initiatives  are  intended  to  operate  within  DOD’s 
existing  organizational  and  process  framework  and  may  not  allow  for 
sufficient  authority  and  control  over  resources  to  effectively  influence 
weapon  system  investments. 

To  improve  DOD’s  ability  to  deliver  a  balanced  mix  of  weapon  system 
programs  at  the  right  time  and  right  cost,  we  are  recommending  the 
department  establish  an  integrated,  portfolio-based  approach  to 
investments  that  incorporates  best  practices  of  successful  commercial 
companies.  To  ensure  the  success  of  such  an  approach,  we  are  also 
recommending  that  DOD  establish  a  single  point  of  accountability  at  the 
department  level  with  the  responsibility,  authority,  and  accountability  for 
ensuring  that  portfolio  management  for  weapon  system  investments  is 
effectively  implemented  across  the  department.  DOD  concurred  with  the 
majority  of  our  recommendations  and  partially  concurred  with  two. 
Generally,  in  responding  to  these  recommendations,  DOD  stated  that  it  is 
undertaking  several  initiatives  and  pilot  efforts  to  improve  the 
department’s  approach  to  investment  and  program  decision  making,  and 
that  implementation  of  any  new  business  rules  will  be  contingent  upon  the 
outcome  of  these  initiatives.  However,  we  believe  the  department’s 
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Background 


current  initiatives  do  not  fundamentally  change  DOD’s  service-centric 
framework  or  sufficiently  integrate  its  decision-making  processes.  DOD 
did  not  provide  comments  regarding  our  recommendation  that  the 
Secretary  establish  a  single  point  of  accountability. 


DOD’s  programs  for  acquiring  major  weapon  systems  have  taken  longer, 
cost  more,  and  delivered  fewer  quantities  and  capabilities  than  planned. 
We  have  documented  these  problems  for  decades.  Most  recently,  we 
reported  that  27  major  weapon  programs  we  have  assessed  since  they 
began  product  development  have  experienced  cost  increases  of  nearly 
34  percent  over  their  original  research,  development,  test,  and  evaluation 
(RDT&E)  estimates,  and  increases  of  almost  24  percent  in  acquisition 
cycle  time  (see  table  l).6 


Table  1 :  Cost  and  Cycle  Time  Growth  for  27  Weapon  Systems 

Billions  of  constant  2007  dollars 

First  full  estimate 

Latest  full  estimate 

Percentage  change 

Total  cost 

$506.4 

$603.1 

19.1 

RDT&E  cost 

$104.7 

$139.7 

33.5 

Weighted  average  acquisition  cycle  time3 

137.9  months 

170.2  months 

23.5 

Source:  GAO  analysis  of  DOD  data. 

“This  is  a  weighted  estimate  of  average  acquisition  cycle  time  for  the  27  programs  based  on  total 
program  costs  at  the  first  full  and  latest  estimates.  The  simple  average  for  these  two  estimates  was 
98.9  months  for  the  first  full  estimate  and  124.6  months  for  the  latest  estimate  resulting  in  a 
26.1  percent  change. 

When  cost  and  schedule  problems  occur  in  one  program,  DOD  often 
attempts  to  pay  for  the  poorly  performing  program  by  taking  funds  from 
others.  Doing  so  has  destabilized  other  programs  and  reduced  the  overall 
buying  power  of  the  defense  dollar  as  DOD  and  the  military  services  are 
forced  to  cut  back  on  planned  quantities  or  capabilities  to  stay  within 
budget  limitations.  The  F-22A  Raptor  program  is  a  case  in  point:  As  costs 
escalated  in  the  program,  the  number  of  aircraft  the  Air  Force  planned  to 
buy  was  drastically  reduced  from  648  to  183.  Similarly,  as  the  Joint 
Tactical  Radio  System  (JTRS)  encountered  development  problems,  the 
number  of  requirements  was  reduced  or  deferred  by  about  one-third.  As  a 
result,  several  programs  that  were  dependent  on  JTRS  also  had  to  make 


5GAO,  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs, 
GAO-07-406SP  (Washington,  D.C.:  Mar.  30,  2007). 
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adjustments  and  go  forward  with  alternative,  less  capable  solutions. 
DOD’s  approach  to  managing  weapon  system  investments  ultimately 
results  in  less  funding  being  available  for  other  competing  needs  in  DOD 
as  well  as  other  federal  priorities,  as  the  expenditure  of  tax  dollars  within 
DOD  reduces  the  amount  of  funding  available  for  those  priorities. 

Taking  into  account  the  differences  between  commercial  product 
development  and  weapons  acquisitions,  we  have  recommended  that  DOD 
adopt  a  knowledge-based,  incremental  approach  to  developing  and 
producing  weapon  systems.  This  type  of  an  approach  requires  program 
officials  to  demonstrate  that  critical  technologies  are  mature,  product 
designs  are  stable,  and  production  processes  are  in  control  at  key 
junctures  in  the  acquisition  process.6 

DOD  has  three  major  processes  involved  in  making  weapon  system 
investment  decisions.  These  processes,  depicted  in  figure  1,  are  the  Joint 
Capabilities  Integration  and  Development  System  (JCIDS),  for  identifying 
warfighting  needs;  the  Planning,  Programming,  Budgeting  and  Execution 
(PPBE)  system,  for  allocating  resources;  and  the  Defense  Acquisition 
System  (DAS),  for  managing  product  development  and  procurement. 
Much  of  our  prior  work  has  focused  on  identifying  commercial  best 
practices  that  could  be  used  to  improve  the  Defense  Acquisition  System — 
from  the  point  just  before  product  development  starts  onward.  In  this 
report,  however,  we  look  at  earlier  stages  in  DOD’s  investment  process — 
from  the  point  where  gaps  in  warfighting  capability  are  assessed  in  JCIDS 
through  the  point  where  alternative  solutions  to  resolve  those  gaps  are 
analyzed  under  the  DAS  (see  fig.  1). 


6GAO,  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early  Improves 
Acquisition  Outcomes,  GAO-02-701  (Washington,  D.C.:  July  15,  2002);  Best  Practices: 
Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better  Weapon  System  Outcomes, 
GAO-Ol-288  (Washington,  D.C.:  Mar.  8,  2001);  and  Best  Practices:  Better  Management  of 
Technology  Development  Can  Improve  Weapon  System  Outcomes,  GAO/NSIAD-99-162 
(Washington,  D.C.:  July  30,  1999). 
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Figure  1 :  DOD’s  Weapon  System  Investment  Process 
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Successful  Companies 
Take  a  Disciplined, 
Integrated  Approach 
to  Prioritize  Market 
Needs  and  Initiate  a 
Balanced  Mix  of 
Executable 
Development 
Programs 


To  ensure  they  achieve  a  balanced  mix  of  executable  development 
programs,  the  successful  commercial  companies  we  reviewed  use  a 
disciplined  and  integrated  approach  to  prioritize  market  needs  and 
allocate  resources.  This  approach,  known  as  portfolio  management, 
requires  companies  to  view  each  of  their  investments  from  an  enterprise 
level  as  contributing  to  the  collective  whole,  rather  than  as  independent 
and  unrelated.  With  this  enterprise  viewpoint,  companies  can  effectively 
(1)  identify  and  prioritize  market  opportunities  and  (2)  apply  available 
resources  to  potential  products  to  select  the  best  mix  of  products  to 
exploit  the  highest  priority — or  most  promising — opportunities. 
Ultimately,  each  of  the  companies  we  reviewed  seeks  to  achieve  a 
balanced  portfolio  that  maximizes  the  return  on  investments  and  moves 
the  company  toward  achieving  its  strategic  goals  and  objectives.  This  type 
of  approach  depends  on  strong  governance  with  committed  leadership, 
clearly  aligned  responsibility,  and  effective  accountability  at  all  levels  of 
the  organization. 
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As  depicted  in  figure  2,  a  portfolio  management  approach  begins  with  an 
enterprise-level  identification  and  definition  of  market  opportunities  and 
then  the  prioritization  of  those  opportunities  within  resource  constraints. 
Once  opportunities  have  been  prioritized,  companies  draft  initial  business 
cases  for  alternative  product  ideas  that  could  be  developed  to  exploit  each 
of  the  highest  priority  opportunities.  Each  alternative  product  proposal — 
represented  by  a  black  dot — enters  a  gated  review  process.  At  each  review 
gate,  product  proposals  are  assessed  against  corporate  resources, 
established  criteria,  competing  products,  and  the  goals  and  objectives  of 
the  company  as  a  whole.  As  alternatives  pass  through  each  review  gate, 
the  number  is  expected  to  decrease,  until  only  those  alternatives  with  the 
greatest  potential  to  succeed  make  it  into  the  product  portfolio. 
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Figure  2:  Portfolio  Management  Approach  to  Product  Investments 


Source:  GAO  analysis  and  presentation  of  commercial  best  practices. 


Identifying  and  Prioritizing 
Market  Opportunities  Lays 
the  Foundation  for 
Achieving  the  Right  Mix  of 
Products 


To  make  informed  decisions  about  what  market  opportunities  to  target, 
the  companies  we  reviewed  first  establish  a  strategy  that  lays  out  the 
overall  goals,  objectives,  and  direction  for  the  company.  As  part  of  their 
strategy,  companies  identify  enterprise-level  sales  and  profit  targets, 
strategic  business  areas  they  want  to  focus  on,  the  extent  to  which  current 
products  and  new  development  efforts  will  support  their  growth 
objectives,  and  how  they  will  allocate  resources  across  business  units  and 
functional  areas.  This  strategy  provides  a  framework  for  the  companies’ 
investment  decisions.  Within  this  framework,  companies  conduct  a  series 
of  market  analyses  to  develop  a  comprehensive  understanding  of  the 
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market  environment,  including  product  trends,  technology  trends,  and 
customer  needs. 


IBM  for  example,  follows  a  structured  market  planning  process  to  identify, 
prioritize,  and  target  attractive  market  segments.  The  first  phase  of  this 
process,  called  Market  Definition,  focuses  on  understanding  the 
marketplace,  including  identifying  potential  customers  and  their  needs.7 
During  this  phase,  IBM  examines  the  marketplace  and  technology 
environments  and  identifies  attractive  market  segments  that  contain 
potential  market  opportunities — where  customer  wants  or  needs  exist. 
Each  segment  is  categorized  into  one  of  four  areas  based  on  needs  of  the 
customers  and  the  company’s  product  offerings  (see  fig.  3):  “strike  zone,” 
“traditional,”  “pushing  the  envelope,”  and  “white  space.”  The  strike  zone 
represents  IBM’s  core  business — market  segments  where  IBM  has  an 
established  customer  base  that  it  is  successfully  serving  with  existing 
product  offerings.  In  contrast,  white  space  represents  market  segments  of 
new  customers  with  wants  and  needs  that  are  new  and  different  for  IBM. 
White  space  opportunities  often  require  discovery  and  innovation.  The 
traditional  and  pushing  the  envelope  areas  fall  between  the  strike  zone  and 
white  space.  Traditional  opportunities  exist  when  new  customers  could  be 
attracted  to  an  existing  market — one  IBM  is  already  active  in — by 
modifying  or  enhancing  existing  products  or  services.  Pushing  the 
envelope  opportunities  exist  where  the  needs  of  current  customer  groups 
move  them  into  a  new  market  segment.  These  attractive  market  segments 
are  prioritized  during  the  next  phase  of  IBM’s  process,  known  as  the 
Capability  Assessment  phase.  During  this  phase  each  segment’s  overall 
attractiveness  and  potential  profitability  are  assessed,  along  with  IBM’s 
available  resources — like  capital,  cash,  and  current  products — and  its 
competitive  position  within  each  segment.  This  analysis  leads  to  the 
selection  of  targeted  market  segments. 


'IBM  focuses  on  providing  its  customers  with  what  it  calls  “industry  integrated  solutions.” 
As  a  result,  portfolios  within  IBM  are  structured  around  the  customers’  and  buyers’ 
behaviors  and  needs  and  not  on  specific  product  offerings.  In  other  words,  the  company 
focuses  on  providing  functional  solutions  to  the  customers,  which  could  be  done  with  a 
number  of  product  offerings  and  services,  and  not  simply  limiting  themselves  to  narrowly 
focused,  single  product  solutions  that  the  customer  can  take  or  leave,  an  approach  that 
IBM  believes  caused  problems  for  the  company  in  the  past. 
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Figure  3:  IBM  Market  Segmentation 
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Source:  IBM’s  IPD  process  (data);  GAO  (analysis  and  presentation). 


Motorola  emphasizes  the  importance  of  targeting  the  right  market 
segments  at  the  enterprise  level  to  ensure  that  a  balanced  mix  of  project 
and  resource  investments  is  maintained.  Officials  noted  that  excessively 
focusing  on  segments  that  require  new  and  innovative  products  can  result 
in  long  cycle  times,  wasted  money,  and  lost  opportunities  elsewhere. 
Likewise,  critical  opportunities  can  be  lost  when  too  much  emphasis  is 
placed  on  simply  continuing  to  invest  in  old  markets  with  old  products. 
According  to  the  officials  we  spoke  with,  the  current  investment  mix  for 
Motorola’s  Government  and  Enterprise  Mobility  Solutions  business  unit  is 
roughly  70-20-10,  where  70  percent  of  its  projects  and  resources  are 
dedicated  to  maintaining  its  core  business,  while  20  percent  are  invested 
in  pursuing  new  markets  with  existing  products  or  introducing  new  or 
enhanced  products  into  existing  markets,  and  the  remaining  10  percent  are 
dedicated  to  discovering  new  markets  and  new  products. 

As  part  of  their  market  analyses,  companies  increasingly  refine  their 
understanding  of  who  their  customers  are  and  what  they  need.  For  several 
of  the  companies  we  met  with,  determining  the  needs  of  their  customers  is 
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complex  because  they  have  multiple  groups  of  customers  to  consider.  For 
example,  Eli  Lilly  has  four  customer  groups  with  diverse  needs:  patients, 
doctors,  insurance  companies,  and  government  regulators.  This 
complexity  is  compounded  when  considering  that  success  in  a  worldwide 
market  is  critically  dependent  on  a  company’s  ability  to  operate  within 
different  governmental  systems,  laws,  and  regulations;  and  regional 
markets.  Several  of  the  companies  we  reviewed  use  a  variety  of  methods — 
including  interviews,  surveys,  focus  groups,  and  concept  tests — to  actively 
engage  their  customers  and  help  determine  what  they  need.  Some 
companies  also  observe  customer  behaviors  to  identify  unstated  wants 
and  needs  that  if  met — assuming  corporate  knowledge  and  resources 
allow — could  actually  exceed  customer  expectations.  While  companies 
actively  seek  customer  input  to  identify  products  that  show  the  most 
promise  and  satisfy  customer  needs,  customers  generally  do  not  identify 
specific  products  to  be  developed. 


Companies  Follow  a 
Disciplined  Process  to 
Identify  New  Products  and 
Achieve  a  Balanced 
Portfolio 


Once  companies  have  identified  and  prioritized  their  market  opportunities, 
they  follow  a  disciplined  process  to  assess  the  costs,  benefits,  and  risks  of 
potential  product  alternatives  and  allocate  resources  to  achieve  a  balanced 
portfolio  that  spreads  risk  across  products,  aligns  with  the  company’s 
strategic  goals  and  objectives,  and  maximizes  the  company’s  return  on 
investment.  At  an  early  stage,  each  alternative  product  is  expected  to  be 
accompanied  by  an  initial  business  case  that  contains  knowledge-based 
information  on  strategic  relevance  and  estimates  of  cost,  technology 
maturity,  and  the  cycle  time  for  getting  the  product  from  concept  to 
market.  To  ensure  comparability  across  alternatives,  companies  require 
initial  business  case  information  to  be  developed  in  a  transparent  manner, 
to  use  specific  standards,  and  to  report  estimates  within  certain  levels  of 
confidence  or  allowable  deviations.  Each  of  the  companies  we  reviewed 
also  stressed  the  importance  of  having  multiple  management  review 
points,  or  gates,  at  early  phases  to  assess  and  prioritize  alternative 
products.  As  products  move  through  review  gates,  from  ideas,  to  more 
concrete  concepts,  to  the  start  of  development  where  a  final  business  case 
is  made,  companies  expect  uncertainties — which  are  typically  inherent  in 
the  early  phases — to  be  addressed  and  estimates  to  become  more  precise. 
Consequently,  the  number  of  viable  alternatives  tends  to  decrease  at  each 
review  gate  as  those  with  the  lowest  potential  for  success  and  least  value 
are  terminated  or  deferred,  while  those  that  are  poised  to  succeed  and 
providing  the  best  value  are  approved  to  proceed  (illustrated  in  fig.  2). 
Companies  emphasized  that  making  tough  go/no-go  decisions  is  critical  to 
keeping  a  balanced  portfolio.  Over  time,  as  potential  new  products  are 
identified,  companies  review  them  against  other  product  investments 
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(proposed  and  existing)  and  rebalance  their  portfolios  based  on  those  that 
add  the  most  value. 

The  companies  we  visited  each  follow  a  disciplined,  gated  review  process 
to  ensure  that  they  commit  to  development  programs  that  help  balance  the 
portfolio  and  that  are  executable  given  available  corporate  resources.  This 
allows  companies  to  avoid  committing  to  more  programs  than  their 
resources  can  support  and  ensure  stability  in  the  programs  they  invest  in. 
Although  the  number  of  review  gates  prior  to  the  start  of  full-scale  product 
development  varied  between  companies — ranging  from  four  at  Procter  & 
Gamble,  to  eight  at  Motorola — they  all  required  potential  products  to 
follow  an  established,  disciplined  process  and  meet  specified  criteria  at 
each  review  point.  For  example,  Caterpillar  assesses  product  alternatives 
at  four  review  gates  prior  to  the  start  of  development — three  of  which 
were  recently  added  to  enhance  the  rigor  of  its  investment  decision 
making.  Each  alternative  must  be  supported  by  a  draft  business  case  that 
includes  quantifiable  data  that  can  be  compared  with  specific  standards 
and  used  to  determine  if  the  related  product  can  move  past  that  gate.  At 
each  gate,  alternatives  are  reviewed  to  ensure  that  knowledge  about 
critical  technologies,  life-cycle  costs,  product  reliability,  and  product 
affordability  is  being  acquired  and  that  the  product  contributes  to 
achieving  the  company’s  strategic  goals  and  objectives. 

Because  developing  a  new  drug  is  costly  and  time  consuming,  Eli  Lilly 
requires  that  the  data  supporting  potential  new  drugs  must  meet  high 
standards  to  ensure  that  managers  are  informed  to  make  sound  investment 
decisions.8  Each  potential  new  drug  must  be  supported  by  an  initial 
business  case  that  contains  information  about  safety  and  efficacy; 
forecasted  revenue;  expected  unit  demand;  capital,  medical,  supply  and 
material,  development,  and  selling  and  marketing  expenses;  and  general 
administrative  costs.  The  initial  business  case  must  also  identify  critical 
success  factors,  state  the  probability  of  technical  success,  and  provide  a 
timeline  that  details  when  major  milestone  events  are  expected  and  how 
long  it  will  take  to  get  the  associated  drug  to  market.  Eli  Lilly  assesses, 
approves,  and  funds  proposed  new  drugs  incrementally.  At  each  milestone 
review  a  contract  is  established  between  the  project  team  and  a 
gatekeeper  committee,  which  contains  deliverables,  time  frames,  and  the 


According  to  Eli  Lilly  representatives,  the  average  cost  to  discover  and  develop  a  new  drug 
ranges  from  $800  million  to  $1  billion,  and  the  average  length  of  time  to  get  a  new  drug 
from  discovery  to  the  market  is  10  to  15  years. 
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costs  to  get  to  the  next  milestone.  Once  this  contractual  agreement  is 
reached  the  budget  is  allocated  for  the  entire  phase.  The  gatekeeper 
committees  expect  each  new  drug  proposal  to  achieve  an  80-percent 
confidence  level  in  their  cost  and  schedule  estimates  for  the  next  phase. 
This  high  level  of  confidence  is  achievable  in  large  part  because  final 
budget  estimates  are  not  developed  by  project  teams  until  2  months  prior 
to  the  milestone  review.  Projects  are  terminated  at  early  points  in  the 
review  process  when  it  is  determined  that  their  critical  success  factors 
cannot  be  achieved.  Because  Eli  Lilly’s  projects  typically  have  a  high 
degree  of  technical  risk,  only  about  1  percent  of  those  that  start  early 
development  actually  make  it  to  the  marketplace. 

Motorola  officials  also  emphasized  the  importance  of  having  sound 
information  when  assessing  potential  new  products.  They  noted  that  a 
process  without  sound  information  will  not  produce  good  outcomes. 
Therefore,  Motorola’s  Government  and  Enterprise  Mobility  Solutions 
business  unit  expects  potential  products  to  be  supported  by  initial 
business  cases  containing  data  that  meet  specific  standards  and  levels  of 
confidence  at  each  review  gate.  For  example,  cost  estimates  for  potential 
products  are  developed  in  several  phases  and  are  expected  to  increase  in 
confidence  with  each  successive  phase.  Early  in  the  investment  planning, 
when  an  initial  business  case  is  first  drafted,  the  confidence  parameters 
are  generous,  ranging  from  as  much  as  75  percent  higher  to  as  much  as 
25  percent  lower  than  what  the  project  will  likely  end  up  costing.  By  the 
time  a  product  alternative  reaches  the  beginning  of  product  development, 
when  a  final  business  case  is  made,  Motorola  expects  the  cost  estimates  to 
be  at  confidence  levels  of  10  percent  higher  and  5  percent  lower.  Proposed 
products  that  fail  to  meet  the  specified  criteria  at  early  review  gates  are 
either  terminated  or  sent  back  to  further  mature  and  reenter  the  review 
process  from  the  beginning. 

The  companies  we  reviewed  use  a  variety  of  portfolio  management  tools 
and  methods  to  inform  the  investment  and  resource  allocation  decisions 
they  make  at  each  review  gate.  Some  companies  employ  scoring  methods, 
using  experts  to  rate  products  based  on  a  number  of  factors — such  as 
strategic  fit,  risk,  and  economic  value — and  use  that  information  to 
prioritize  alternative  products.  Another  common  tool  plots  alternative 
products  on  a  decision  matrix  that  compares  factors  such  as  costs  and 
benefits,  or  risks  and  rewards  of  competing  alternatives.  Using  this  type  of 
matrix,  alternative  products  are  often  represented  by  circles,  where  the 
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size  of  the  circles  provides  information  about  key  constraints  such  as 
available  annual  resources  or  the  estimated  annual  costs  for  each 
alternative.  For  example,  figure  4  compares  risk  and  expected  rewards9  by 
plotting  competing  alternatives  on  a  matrix.  Alternatives  that  fall  into  the 
upper  left  quadrant  are  high  risk  and  low  reward,  while  alternatives  that 
fall  into  the  lower  right  quadrant  are  low  risk  and  high  reward.  By 
weighing  risk  against  rewards  and  considering  constraints  such  as  annual 
resources  or  annual  cost,  this  tool  provides  critical  information  and  a 
structured  means  to  help  managers  make  informed  decisions.  Company 
officials  at  Procter  &  Gamble  emphasized  the  importance  of  selecting  a 
balanced  mix  of  products  to  pursue.  They  noted  that  pursuing  only  low- 
risk  and  high-reward  products  at  the  expense  of  more  innovative,  higher- 
risk  products  could  cause  the  company  to  miss  out  on  opportunities  to 
improve  their  competitive  standing  in  the  marketplace.  Likewise, 
excessive  pursuit  of  higher-risk  products  with  the  potential  for  high 
returns  could  also  result  in  lost  opportunities  to  elsewhere. 


Companies  commonly  measure  rewards  and  benefits  using  cash  flow  analysis  known  as 
net  present  value  (NPV)  analysis.  NPV  techniques  can  show,  in  today’s  dollars,  the  relative 
net  cash  flow  of  various  alternatives  over  a  long  period  of  time.  Simply  stated,  net  cash 
flow  is  the  amount  of  dollars  that  is  left  after  sales  and  revenues  have  offset  expenses.  In 
general,  the  greater  the  net  cash  flow  for  a  particular  investment,  the  greater  the  return  on 
the  investment. 
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Figure  4:  Risk  Versus  Rewards  Matrix 
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Source:  GAO. 


Recognizing  the  inherent  risks  in  pursuing  a  new  development  program — 
that  overruns  or  underruns  in  one  business  case  result  in  lost  opportunity 
to  invest  resources  in  another  worthwhile  project — IBM  permits  products 
to  deviate  from  their  original  business  case  estimates  as  long  as  the 
deviation  is  within  established  limits.  These  limits  are  specified  in  a 
contractual  document  resulting  from  negotiations  between  senior 
management  and  project  managers  and  signed  at  the  beginning  of  product 
development.  Product  development  teams  are  expected  to  execute 
according  to  the  contract;  if  established  thresholds  are  breached,  action  is 
taken  immediately  to  reassess  the  product  within  the  context  of  the 
portfolio  and  determine  whether  it  is  still  a  relevant  and  affordable 
investment  to  pursue. 
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Successful  Portfolio 
Management  Requires 
Strong  Governance  with 
Committed  Leadership, 
Empowered  Decision 
Makers,  and  Effective 
Accountability 


Successful  portfolio  management  requires  strong  governance  with 
committed  leadership  that  empowers  portfolio  managers  to  make 
decisions  about  the  best  way  to  invest  resources  and  holds  those 
managers  accountable  for  the  outcomes  they  achieve.  The  companies  we 
reviewed  indicated  that  it  is  critical  to  have  commitment  from  the  top 
leaders  of  the  organization  and  recognition  at  all  levels  that  what  is  best 
for  the  company  must  be  a  priority,  and  not  simply  what  is  best  for  a 
particular  business  unit  or  product  line.  In  addition,  the  companies 
emphasized  that  roles  and  responsibilities  for  implementing  portfolio 
management,  including  the  designation  of  who  is  responsible  for  product 
investment  decisions  and  oversight,  must  be  clearly  defined.  Because 
portfolio  managers  are  on  the  front  line,  the  companies  we  reviewed 
empower  these  managers  to  make  product  investment  decisions  and  hold 
them  accountable  for  outcomes,  not  just  for  individual  products  but  also 
for  the  overall  performance  of  their  portfolios.  To  support  their  portfolio 
managers,  the  companies  encourage  collaboration  and  communication, 
including  sharing  bad  news  early.  Several  companies  also  emphasized  the 
importance  of  supporting  their  portfolio  managers  with  cross-functional 
teams,  composed  of  representatives  from  the  key  functional  areas  within 
the  company — such  as  science  and  technology,  marketing,  engineering, 
and  finance — to  ensure  that  they  are  adequately  informed  when  making 
investment  decisions.  To  ensure  accountability,  companies  often  use 
incentives  and  disincentives,  including  promotion  and  termination.  We 
have  previously  reported  that  high-performing  organizations  have 
monetary  and  other  rewards  that  clearly  link  employee  knowledge,  skills, 
and  contributions  to  achieving  the  organization’s  goals  and  objectives.10 
These  organizations  underscore  the  importance  of  holding  individuals 
accountable  and  aligning  performance  expectations  with  organizational 
goals  and  cascade  those  expectations  down  to  lower  levels.  Companies 
stressed  that  the  transformation  to  portfolio  management  takes  time  and 
requires  not  only  process  changes  but  cultural  changes  throughout  the 
company. 


Eli  Lilly  emphasized  that  a  key  to  making  its  portfolio  management 
process  work  is  having  a  single  committee  with  a  high-level  official  in 
charge  responsible  for  making  product  investment  decisions.  Previously, 
the  company  had  a  multi-layered  committee  structure  in  place,  and 
decisions  were  made  based  on  reaching  a  consensus — an  approach  that 


10GAO,  Results-Oriented  Cultures:  Creating  a  Clear  Linkage  between  Individual 
Performance  and  Organizational  Success,  GAO-03-488  (Washington,  D.C.:  Mar.  14,  2003). 
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was  viewed  as  cumbersome  and  lengthy.  Eli  Lilly  also  ensures 
accountability  by  directly  linking  management  and  employee  bonuses  to 
the  overall  success  of  the  company.  Individual  employee  performance 
objectives  are  aligned  with  specific  company  objectives,  such  as  meeting 
budgetary  goals,  time  frames,  and  data  quality  levels  for  a  given  project. 
Achievement  of  individual  employee  objectives  is  measured  periodically  to 
provide  feedback  to  the  employee.  Eli  Lilly  officials  stressed  that  having 
the  right  performance  metrics  in  place  is  important  because  ultimately  you 
get  what  you  measure;  therefore,  be  sure  to  measure  the  right  things. 

Motorola  considers  accountability  to  be  the  critical  factor  in  making  its 
portfolio  management  process  successful.  In  addition,  Motorola’s  culture 
is  not  averse  to  reporting  bad  news  to  management.  Project  managers  are 
encouraged  to  report  problems  early  so  that  they  can  be  addressed  before 
they  get  out  of  control.  Senior  managers,  however,  are  not  intimately 
involved  in  the  day-to-day  decision  making  for  individual  products.  That 
responsibility,  in  nearly  every  case,  is  delegated  to  the  business  unit 
general  manager.  The  general  manager  of  a  business  unit  is  held 
accountable  for  ensuring  that  the  products  within  his  unit  succeed  at  all 
levels.  The  general  manager  is  responsible  for  holding  product  managers 
accountable  for  the  attainment  of  critical  knowledge  at  key  points  and  the 
performance  of  their  individual  products  overall.  General  managers  and 
product  managers  can  be  fired  for  not  meeting  objectives.  Motorola 
believes  that  if  managers  are  held  accountable  for  results,  then  they  have 
more  desire  to  get  it  right. 


Lacking  an  Integrated, 
Portfolio-Based 
Approach,  DOD  Has 
Too  Many  Programs 
Competing  for 
Limited  Resources 


Although  the  military  services  fight  together  on  the  battlefield  as  a  joint 
force,  they  do  not  identify  warfighting  needs  and  make  weapon  system 
investment  decisions  together.  DOD  has  taken  steps  to  identify  warfighting 
needs  through  a  more  joint  requirements  process,  but  the  department’s 
service-centric  structure  and  fragmented  decision-making  processes  are  at 
odds  with  the  integrated,  portfolio  management  approach  used  by 
successful  commercial  companies  to  make  enterprise-level  investment 
decisions.  Consequently,  DOD  has  less  assurance  that  its  weapon  system 
investment  decisions  address  its  most  important  warfighting  needs  and  are 
affordable  in  the  context  of  its  overall  fiscal  resources.  In  addition,  DOD 
commits  to  products  earlier  than  the  companies  we  reviewed  and  with  far 
less  knowledge  about  their  cost  and  feasibility.  This  leads  to  poor  program 
outcomes  and  funding  instability,  as  the  department  attempts  to  fix 
troubled  programs  by  taking  funds  from  others. 
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Service-centric  Structure 
and  Fragmented  Decision¬ 
making  Processes  Impede 
DOD’s  Ability  to  Prioritize 
Warfighting  Needs 


Although  recent  DOD  policy  emphasizes  a  more  joint  approach  to 
identifying  and  prioritizing  warfighting  needs,11  DOD’s  service-centric 
structure  and  fragmented  decision-making  processes  hinder  the  policy’s 
successful  implementation.  This  policy,  which  introduced  the  JCIDS 
process,  calls  for  a  wider  range  of  stakeholders  than  before,  including 
more  customer  (i.e.,  combatant  command)  involvement;  introduces  new 
methodologies  intended  to  foster  jointness;  and  groups  warfighting  needs 
into  eight  functional  areas  based  on  warfighting  capabilities — such  as 
netcentric,  force  application,  and  battlespace  awareness12 — that  cut  across 
the  military  services  and  defense  agencies.  The  JCIDS  process  emphasizes 
early  attention  to  the  fiscal  implications  of  newly  identified  needs, 
including  identifying  ways  to  pay  for  new  capabilities  by  divesting  the 
department  of  lower  priority  or  redundant  capabilities.  Despite  these 
provisions,  assessments  of  warfighting  needs  continue  to  be  driven  by  the 
services  and  to  be  based  on  investment  decision-making  processes  that  do 
not  function  together  to  ensure  that  DOD  pursues  needs  that  its  resources 
can  support. 


The  military  services  identify  warfighting  needs  individually,  and 
department-level  organizations  are  not  optimized  to  integrate  the  services’ 
results  or  evaluate  their  fiscal  implications  early  on.  Historically,  this 
approach  has  contributed  to  duplication  in  weapon  systems  and 
equipment  that  does  not  interoperate.  At  the  department  level,  Functional 
Capability  Boards  oversee  each  of  the  eight  functional  areas,  reviewing  the 
services’  assessments,  and  providing  recommendations  to  the  Joint 
Requirements  Oversight  Council  (JROC),  which  leads  the  JCIDS  process. 
However,  defense  experts  and  DOD  officials  report  that  the  Functional 
Capability  Boards  do  not  have  the  staff  or  analytical  resources  required  to 
effectively  evaluate  service  assessments  within  the  context  of  the  broader 
capability  portfolio  and  assess  whether  the  department  can  afford  to 
address  a  particular  capability  gap.  Several  recent  studies  have 
recommended  that  DOD  increase  joint  analytical  resources  for  a  less 


^Chairman  of  the  Joint  Chiefs  of  Staff  Instruction,  Joint  Capabilities  Integration  and 
Development  System,  CJCSI  3170. 01E  (May  11,  2005).  The  original  instruction  was  CJCSI 
3170.01C  (June  24,  2003). 

12The  other  capability  areas  are  command  and  control,  focused  logistics,  force 
management,  force  protection,  and  joint  training. 
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stovepiped  understanding  of  warfighting  needs.13  In  addition,  the  boards 
lack  the  authority  to  allocate  resources  and  to  make  or  enforce  decisions 
to  divest  their  capability  area  of  existing  programs  to  pay  for  new  ones — 
authority  successful  companies  provide  to  their  portfolio  managers. 

Finally,  some  defense  experts  contend  that  the  service  ties  of  JROC’s 
members — that  is,  the  services’  Vice  Chiefs  and  the  Assistant  Commandant 
of  the  Marine  Corps — reinforce  service  stovepipes.  To  better  ensure  a 
more  joint  perspective,  they  recommend  a  more  diverse  JROC,  with 
representatives  from  other  department-level  organizations  and  the 
combatant  commands.14 

Resource  allocation  decisions  are  made  through  a  separate  process — the 
Planning,  Programming,  Budgeting,  and  Execution  system  (PPBE) — which 
hinders  the  department’s  ability  to  weigh  the  relative  costs,  benefits,  and 
risks  of  investing  in  new  weapon  systems  early  on.  Within  the  PPBE 
system,  the  individual  military  services  are  responsible  for  budgeting  and 
allocating  resources  under  authority  that  is  commonly  understood  to  be 
based  on  Title  10  of  the  United  States  Code.16  PPBE  is  structured  by 
military  service  and  defense  program,  although  the  department  integrates 
data  on  the  services’  current  and  projected  budget  requests  under 
11  crosscutting  mission  areas  called  Major  Force  Programs.  The  cross¬ 
cutting  view  provided  by  the  Major  Force  Program  structure  is  intended  to 
facilitate  a  strategic  basis  for  resource  allocation,  allowing  the  Secretary  of 
Defense  to  more  easily  see  where  the  greatest  mission  needs  are  and  to 
re-allocate  funds  to  meet  those  needs  regardless  of  which  service  stands  to 
gain  or  lose.  However,  we  have  reported  in  the  past  that  the  Major  Force 


13Institute  for  Defense  Analyses,  Improving  Integration  of  Department  of  Defense 
Processes  for  Capabilities  Development  and  Planning  (Sept.  2006);  Center  for  Strategic 
and  International  Studies,  Beyond  Goldwater-Nichols:  Defense  Reform  for  a  New  Strategic 
Era,  Phase  1  Report  (Mar.  2004);  and  Joint  Defense  Capabilities  Study  Team  (DOD),  Joint 
Defense  Capabilities  Study:  Improving  DoD  Strategic  Planning,  Resourcing  and 
Execution  to  Satisfy  Joint  Capabilities.  Final  Report  (Jan.  2004). 

14Assessment  Panel  of  the  Defense  Acquisition  Performance  Assessment  Project  for  the 
Deputy  Secretaiy  of  Defense,  Defense  Acquisition  Performance  Assessment  Report  (Jan. 
2006);  Center  for  Strategic  and  International  Studies,  Beyond  Goldwater-Nichols:  U.S. 
Government  and  Defense  Reform  for  a  New  Strategic  Era,  Phase  2  Report  (July  2005);  and 
M.  Thomas  Davis,  “The  JROC:  Doing  What?  Going  Where?”  National  Security  Studies 
Quarterly  (Summer  1998). 

15  Sections  3013,  5013,  and  8013  of  Title  10  grant  authority  to  the  Secretaries  of  the  Army, 
the  Navy,  and  the  Air  Force,  respectively,  to  conduct  all  affairs  of  their  departments, 
including  recruiting,  organizing,  supplying,  equipping,  training,  servicing,  mobilizing, 
demobilizing,  administering,  maintaining,  and  military  construction  and  maintenance. 
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Program  structure  has  not  provided  sufficient  visibility  in  certain  mission 
areas.16  Moreover,  although  they  cut  across  the  services,  the  program 
mission  areas  are  not  consistent  with  the  more  recently  established 
capability  areas  used  in  the  JCIDS  process,17  and  as  a  result,  it  is  difficult  to 
relate  resources  to  capabilities.  For  example,  in  prior  work,  we  observed 
that  the  Major  Force  Programs  contain  large  numbers  of  programs  with 
varied  capabilities,  complicating  comparisons  needed  to  understand 
defense  capabilities  and  associated  trade-off  decisions.18  We  have 
recommended  that  DOD  report  funding  levels  for  defense  capabilities  in 
its  Future  Years  Defense  Program  report  to  the  Congress,  which  is 
currently  organized  by  the  Major  Force  Programs. 

In  addition,  our  analysis  of  DOD’s  investment  accounts — which  pay  for 
developing,  testing,  and  buying  weapon  systems  and  other  equipment — 
indicates  that  DOD  generally  does  not  allocate  resources  on  a  strategic 
basis.  Figure  5  illustrates  that  the  service  allocations  as  a  percentage  of  the 
department’s  overall  investment  budget  have  remained  relatively  static  for 
the  25-year  period  we  examined,  even  though  DOD’s  strategic  environment 
and  warfighting  needs  have  changed  dramatically  during  that  time,  with 
the  demise  of  the  cold  war  and  the  emergence  of  the  global  war  on  terror.19 
In  contrast,  successful  commercial  companies  using  portfolio  management 
would  expect  to  see  their  resource  allocations  across  business  areas  to 
reflect  changes  in  the  marketplace  and  the  competitive  environment. 


16GAO,  Military  Transformation:  Actions  Needed  by  DOD  to  More  Clearly  Identify  Triad 
Spending  and  Develop  a  Long-term  Investment  Approach,  GAO-05-540  (Washington,  D.C.: 
June  30,  2005). 

17The  Major  Force  Programs  are  as  follows:  Strategic  Forces;  General  Purpose  Forces; 
Command,  Control,  Communications,  and  Intelligence;  Mobility  Forces;  Guard  and  Reserve 
Forces;  Research  and  Development;  Central  Supply  and  Maintenance;  Training,  Medical, 
and  other  General  Personnel  Activities;  Administration  and  Associated  Activities;  Support 
of  Other  Nations;  and  Special  Operations  Forces. 

18GAO,  Future  Years  Defense  Program :  Actions  Needed  to  Improve  Transparency  of 
DOD’s  Projected  Resource  Needs,  GAO-04-514  (Washington,  D.C.:  May  7,  2004). 

19The  fiscal  framework  of  the  federal  government  constrains  DOD’s  ability  to  totally  control 
its  investment  allocations.  Ultimately  resource  allocation  decisions  are  made  by  the 
Congress  in  the  annual  authorization  and  appropriations  process. 
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Figure  5:  Service  Allocations  of  DOD’s  Investment  Budget  (FY1986  through  FY2011) 

Percentage 

100  r— - - - - 


90 
80 
70 
60 
50 
40 
30 
20 
10 
0 

1986  1987  1988  1989  1990  1991  1992  1993  1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011 


Defense  wide 


Air  Force 


Fiscal  year 


Source:  DOD  (data):  GAO  (analysis  and  presentation). 


PPBE  and  JCIDS  are  led  by  different  organizations  (see  fig.  6),  as  is  the 
third  of  the  three  processes  involved  in  DOD’s  weapon  system  investment 
decisions,  the  Defense  Acquisition  System  (DAS),  making  it  difficult  to 
hold  any  one  person  or  organization  accountable  for  investment 
outcomes.  The  2006  Quadrennial  Defense  Review  highlighted  the  need  for 
governance  reforms,20  and  a  2006  study  commissioned  by  DOD  observed 
that  the  budget,  acquisition,  and  requirements  processes  are  not 
connected  organizationally  at  any  level  below  the  Deputy  Secretary  of 
Defense,  concluding  that  this  structure  induces  instability  and  erodes 
accountability.21  The  Under  Secretary  of  Defense/Acquisitions, 
Technology,  and  Logistics  (USD/AT&L)  has  stated  that  weapon  system 
investment  decisions  are  a  shared  responsibility,  and,  therefore,  no  one 


20The  2006  Quadrennial  Defense  Review  led  to  the  Institutional  Reform  and  Governance 
project,  which  is  focusing  on  (1)  integrating  core  decision  processes,  (2)  aligning  and 
focusing  the  department’s  governance  and  management  functions  under  an  integrated 
enterprise  model,  as  well  as  (3)  establishing  a  common  and  authoritative  analytical 
framework  to  link  strategic  decisions  to  execution. 

21Assessment  Panel  of  the  Defense  Acquisition  Performance  Assessment  Project  for  the 
Deputy  Secretary  of  Defense,  Defense  Acquisition  Performance  Assessment  Report 
(January  2006). 
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individual  is  accountable  for  these  decisions.  At  a  broader,  strategic  level, 
we  have  stated  in  prior  work  that  DOD  has  lacked  sustained  leadership 
and  accountability  for  various  department-wide  management  reform 
efforts,22  including  the  establishment  of  an  effective  risk  management 
approach  as  a  framework  for  decision  making.23  This  approach  would  link 
strategic  goals  to  plans  and  budgets,  assess  the  value  and  risks  of  various 
courses  of  action  as  a  tool  for  setting  investment  priorities  and  allocating 
resources  at  the  department  level,  and  use  performance  measures  to 
assess  outcomes.  To  address  the  lack  of  sustained  leadership,  we  have 
supported  legislation  to  create  a  chief  management  official  at  DOD.24 


22GAO,  Department  of  Defense:  Sustained  Leadership  Is  Critical  to  Effective  Financial 
and  Business  Transformation,  GAO-06-1000T  (Washington,  D.C.:  Aug.  3,  2006);  and 
Department  of  Defense:  Further  Actions  Are  Needed  to  Effectively  Address  Business 
Management  Problems  and  Overcome  Key  Business  Transformation  Challenges, 
GAO-05-140T  (Washington,  D.C.:  Nov.  18,  2004). 

23GAO,  Defense  Management:  Additional  Actions  Needed  to  Enhance  DOD’s  Risk-Based 
Approach  for  Making  Resource  Decisions,  GAO-06-13  (Washington,  D.C.:  Nov.  15,  2005). 

24S.  780,  109th  Cong.  §1  (2005). 
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Figure  6:  Governance  of  DOD’s  Investment  Process 
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The  Office  of  the  Secretary  of  Defense  (OSD)  does  not  assess  the  funding 
implications  of  a  proposed  program  at  the  front  end  of  the  investment 
process,  when  it  is  initially  validated  by  JROC.  JCIDS  is  a  continuous, 
need-driven  process  that  unfolds  in  response  to  warfighting  needs  as  they 
are  identified.  However,  PPBE  is  a  calendar-driven  process  comprised  of 
phases  that  occur  over  a  2-year  cycle,  thus  OSD’s  formal  review  of  a 
proposed  program  is  not  often  synchronized  with  JROC’s,  and  can  occur 
several  years  later.25  Nevertheless,  according  to  Joint  Staff  and  AT&L 
officials  we  met  with,  proposed  programs  begin  to  gain  momentum  when 
they  are  validated  by  JROC,  and  they  become  very  difficult  to  stop.  These 
officials  indicated  that  momentum  begins  to  gather  because  the  services 


25DAS  is  an  event-driven  process  structured  into  discrete  phases  separated  by  major 
decision  points  called  milestones  that  can  be  tailored  to  individual  programs. 


Page  24 


GAO-07-388  Best  Practices 


start  programming  and  budgeting  for  the  proposed  capability  right  away  to 
secure  funding,  generally  several  years  before  actual  product  development 
begins  and  before  OSD  formally  reviews  the  services’  programming  and 
budgeting  proposals.  In  the  interim,  the  services  have  not  only  budgeted 
for  their  proposed  programs,  but  established  a  program  office,  conducted 
their  Analysis  of  Alternatives,  and  identified  specific  user  requirements. 
OSD’s  programming  and  budgeting  review  occurs  at  the  back  end  of  the 
investment  process,  when  it  is  difficult  and  disruptive  to  make  changes, 
such  as  terminating  existing  programs  to  pay  for  new,  higher  priority 
programs. 

These  practices  have  contributed  to  the  department  starting  more 
programs  than  its  resources  can  support.  DOD  defers  much  of  the 
additional  cost  of  its  programs  into  the  future,  resulting  in  what  some  have 
characterized  as  a  fiscal  bow  wave  (illustrated  in  fig.  7).  This  bow  wave 
has  grown  at  a  pace  that  greatly  exceeds  DOD’s  annual  funding  increases. 
The  cost  remaining  for  DOD’s  major  weapons  programs  increased  almost 
135  percent  between  1992  and  2006,  while  the  department’s  annual  funding 
level  only  increased  57  percent  over  that  same  time  period.  If  this  trend 
goes  unchecked,  Congress  will  likely  be  faced  with  a  difficult  choice:  pull 
funds  from  other  high-priority  federal  programs  to  support  DOD’s 
acquisitions  or  accept  less  warfighting  capability  than  originally  promised. 


Page  25 


GAO-07-388  Best  Practices 


Figure  7:  Costs  Remaining  Versus  Annual  Appropriations  for  Major  Defense  Acquisitions 

Dollars  (in  billions) 

1,200 


Fiscal  year 


Source:  DOD  (data);  GAO  (analysis  and  presentation). 


DOD  Commits  to  a 
Solution  Earlier  and  with 
Less  Knowledge 


DOD  commits  to  a  solution  to  address  a  warfighting  need  earlier  in  the 
investment  process  than  commercial  companies  do  and  before  it  has 
adequate  knowledge  about  cost  and  technical  feasibility.  Proposed  options 
for  resolving  a  gap  in  military  capability  are  submitted  in  an  Initial 
Capabilities  Document  (ICD).  DOD  guidance  states  that  this  document 
should  contain  a  range  of  approaches  based  in  part  on  the  cost  and 
technological  feasibility  posed  by  the  approaches,  laying  the  foundation 
for  a  more  detailed  Analysis  of  Alternatives  to  be  conducted  under  the 
Defense  Acquisition  System.  In  addition,  JROC  is  to  receive  a  briefing  on 
the  ICD  that  follows  a  standard  format  and  addresses  such  issues  as 


•  linkage  of  the  proposal  to  strategic  guidance; 

•  the  time  frame  within  which  the  capability  is  needed; 

•  the  threat/operational  environment; 

•  risks  and  assumptions  (including  the  risk  associated  with 
proceeding  and  not  proceeding  with  solutions  to  each);  and 

•  a  description  of  the  best  materiel  and  non-materiel  approaches 
based  upon  cost,  efficacy,  performance,  technology  maturity,  and 
risk. 
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Although  DOD  guidance  calls  for  the  analysis  of  a  solution’s  cost  and 
feasibility,  we  found  that  ICDs  contained  little  of  this  type  of  information. 
Several  DOD  officials  we  met  with,  who  are  directly  involved  in  the  JCIDS 
process,  did  not  believe  cost  and  feasibility  information  was  mandated  at 
this  point.  In  our  review  of  14  unclassified  ICDs  approved  by  JROC  from 
2003-06,  we  found  that  11  did  not  contain  acquisition  cost  estimates  and 
12  did  not  contain  estimates  of  the  technical  feasibility  of  proposed 
solutions.26  We  also  found  that  JCIDS  guidance  does  not  specify  the  level 
of  accuracy  sought  in  cost  and  feasibility  estimates,  and  a  white  paper  that 
does  provide  recommendations  in  this  regard  is  advisory.27 

We  found  that  ICDs  generally  focused  on  the  strategic,  or  operational, 
relevance  of  proposed  solutions,  but  a  lack  of  guidance  and  an  evolving 
methodology  have  raised  questions  about  the  accuracy  of  data  supporting 
those  assessments.  JCIDS  uses  new  joint  warfighting  concepts28  to 
translate  top-level  military  strategy  into  the  capabilities  a  commander 
might  need  on  the  battlefield.  The  joint  concepts  underpin  a  capabilities- 
based  approach29  to  identifying  requirements,  in  which  analyses  are 
expected  to  focus  on  broad  military  capabilities  rather  than  service- 
specific  platforms.  However,  the  joint  concepts  and  capabilities-based 
assessments  are  works  in  progress.  The  concepts  are  being  updated  due  to 
concerns  about  their  scope,  and  guidance  on  conducting  a  capabilities- 


26The  14  ICDs  we  reviewed  are  related  to  weapon  systems  and  were  finalized  after  June  24, 
2003  (the  publication  date  for  the  initial  JCIDS  instruction).  They  are  unclassified  ACAT  I, 
II,  or  III  ICDs  contained  in  the  Joint  Staffs  Knowledge  Management  Decision  Support  Tool 
database,  which  serves  in  part  as  a  repository  for  JCIDS  documents. 

"‘The  white  paper  suggests  roughly  characterizing  the  20-year-lifecycle  costs  of  proposed 
solutions  in  terms  of  developmental  costs,  facility  or  infrastructure  costs,  per-unit  and 
rough  force-level  acquisition  costs,  and  recurring  operating  costs.  It  states  that  rough 
estimates  of  the  technical  feasibility  of  proposed  solutions  should  be  developed,  not  at  the 
engineering  level  because  of  the  broad  range  of  possibilities,  but  at  the  least  to  characterize 
as  no  risk,  very  low  risk,  low  risk,  medium  risk,  and  high  risk  using  legitimate  technology 
experts.  “Whitepaper  on  Conducting  a  Capabilities-Based  Assessment  Under  the  Joint 
Capabilities  Integration  and  Development  System,”  Joint  Chiefs  of  Staff  J-8/Force 
Application  Assessment  Division  (Jan.  2006). 

"8Joint  future  concepts  are  visualizations  of  future  operations  that  describe  how  a 
commander  might  employ  warfighting  capabilities  to  achieve  effects  and  objectives. 

29The  2001  Quadrennial  Defense  Review  directed  DOD  to  implement  a  capabilities-based 
planning  approach.  Capabilities-based  planning  has  been  described  as  a  framework  for 
defense  planning  and  decision  making  in  a  strategic  environment  characterized  by 
uncertainty  and  a  fiscal  environment  characterized  by  limited  resources.  See  Paul  K.  Davis, 
Analytic  Architectures  for  Capabilities-Based  Planning,  Mission  System  Analysis,  and 
Transformation  (Santa  Monica,  Calif.:  RAND,  2002). 
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based  analysis  has  been  lacking.  Several  DOD  officials  we  met  with  stated 
that  assessments  vary  in  their  rigor,  and  a  senior  Joint  Staff  official  said 
that  training  on  requirements  development  is  one  of  three  central 
challenges  at  present.  In  January  2007,  we  reported  that  DOD  officials 
described  concerns  about  the  analytical  framework  for  a  capabilities- 
based  assessment  on  joint  seabasing,  which  could  lead  to  inaccurately 
identifying  gaps  in  implementing  the  concept.30  Enhancing  a  seabasing 
capability  is  expected  to  be  costly  and  could  be  the  source  of  billions  of 
dollars  of  investment  if  DOD  chooses  an  option  involving  the  development 
of  new  ships. 

DOD  does  not  consistently  follow  a  disciplined  review  process  to  ensure 
that  proposed  solutions  are  making  progress  toward  an  executable 
development  program,  although  DOD  policy  emphasizes  that  such  reviews 
are  necessary.31  DOD’s  policy  identifies  several  key  decision  points  prior  to 
starting  a  new  weapon  system  development  program: 

•  an  initial  decision  point,  where  the  Initial  Capabilities  Document  is 
reviewed,  validated,  and  approved  by  the  JROC; 

•  a  Concept  Decision  review,  where  entry  into  the  concept  refinement 
phase  of  the  Defense  Acquisition  System  should  be  authorized;  and 

•  a  Milestone  A  decision  point,  where  a  preferred  solution  and  a 
technology  development  strategy  should  be  reviewed  and 
approved.32 

Since  Initial  Capabilities  Documents  generally  do  not  contain  information 
on  cost  and  technical  feasibility,  the  JROC  does  not  have  a  sufficient  basis 


30GAO,  Force  Structure:  Joint  Seabasing  Would  Benefit  from  a  Comprehensive 
Management  Approach  and  Rigorous  Experimentation  before  Services  Spend  Billions  on 
New  Capabilities,  GAO-07-211  (Washington,  D.C.:  Jan.  26,  2007).  Joint  seabasing  is  one  of 
several  evolving  concepts  describing  how  commanders  in  the  future  will  project  and 
sustain  forces  for  conducting  joint  military  operations  without  relying  on  immediate  access 
to  nearby  land  bases. 

31CJCSI  3170.01E  ,  Joint  Capabilities  Integration  and  Development  System;  DODI  5000.2, 
Operation  of  the  Defense  Acquisition  System. 

32The  JROC  has  another  decision  point  just  prior  to  program  initiation,  when  it  reviews, 
validates,  and  approves  a  Capabilities  Development  Document.  Approval  and  validation  of 
a  Capabilities  Development  Document  is  a  key  entrance  criteria  for  initiating  a  new 
development  program  at  Milestone  B.  We  did  not  include  the  Capabilities  Development 
Document  decision  point  in  our  current  review  because  it  is  closely  associated  with 
Milestone  B,  the  focus  of  many  of  our  former  reviews.  See  the  list  of  related  GAO  products 
on  the  last  pages  of  this  report. 
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for  making  go/no-go  decisions  at  the  initial  decision  point.  In  the  4  years 
since  JCIDS  was  implemented,  nearly  all  of  the  warfighting  needs 
identified  by  the  services  and  submitted  for  review  in  an  ICD  have  been 
validated  and  sent  into  the  acquisition  pipeline  for  further  analysis  as 
potential  programs,  which  calls  into  question  whether  go/no-go  decisions 
are  the  point  of  this  first  key  gate.  Information  on  cost  and  feasibility  is 
generally  developed  after  the  ICD  is  approved  and  proposed  solutions 
undergo  further  refinement  through  an  Analysis  of  Alternatives  (AO A).  An 
AOA  should  compare  alternative  solutions  in  terms  of  life-cycle  cost, 
schedule,  and  operational  effectiveness,  leading  up  to  the  identification  of 
a  preferred  alternative.  However,  officials  from  PA&E  and  the  Joint  Staff 
indicate  that  AO  As  often  make  a  case  for  a  single  preferred  solution. 
Several  of  them  indicated  other  concerns  about  AO  As,  such  as  not  setting 
up  trade-off  discussions,  lack  of  analytical  rigor,  length,  and  timeliness. 

In  any  case,  the  next  review  points — the  Concept  Decision  and  Milestone 
A — are  often  skipped;  thus,  the  opportunity  to  review  an  evolving  business 
case  and  to  make  go/no-go  decisions  is  bypassed.  In  prior  work,  we  found 
that  80  percent  of  the  programs  we  reviewed  entered  the  Defense 
Acquisition  System  at  Milestone  B  without  holding  any  prior  major 
reviews,  such  as  a  Milestone  A  review.33  Such  reviews  are  intended  to 
provide  acquisition  officials  with  an  opportunity  to  assess  whether 
program  officials  had  the  knowledge  needed  to  develop  an  executable 
business  case.  Senior  officials  with  OSD  confirmed  that  this  is  a  common 
practice  among  defense  acquisition  programs.  We  concluded  that  this 
practice  eliminates  a  key  opportunity  for  decision  makers  to  assess  the 
early  product  knowledge  needed  to  establish  a  business  case  that  is  based 
on  realistic  cost,  schedule,  and  performance  expectations.  In  addition,  we 
found  that  programs  are  regularly  approved  to  begin  development  even 
though  officials  reported  levels  of  knowledge  below  the  criteria  suggested 
in  DOD’s  acquisition  policy. 

There  is,  then,  generally  little  department-level  oversight  between  the 
point  at  which  an  ICD  is  approved  and  when  system-level  requirements  are 
validated  and  product  development  is  initiated.  At  this  point,  as  we 
indicated  earlier,  there  is  generally  no  turning  back,  because  the  services 
have  invested  considerable  time  and  money,  established  a  budget,  and 


33GAO,  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost  and 
Schedule  Problems  under  DOD’s  Revised  Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13, 
2006).  Our  review  focused  on  the  Concept  Decision  and  Milestone  A  reviews  points  and  did 
not  assess  the  extent  to  which  JROC  reviews  were  held. 
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formed  a  constituency  for  a  proposed  program,  and  decision  makers 
become  reluctant  to  terminate  a  program  or  send  it  back  for  further  study. 


DOD  Is  Piloting  Several 
Initiatives  to  Address 
Disconnects  in  Investment 
Decision-making 


In  response  to  the  2006  Quadrennial  Defense  Review  and  other  recent 
acquisition  reform  studies,34  DOD  has  undertaken  several  key,  interrelated 
initiatives  intended  to  strengthen  the  department’s  approach  to  investment 
decisions.  The  initiatives  include  (1)  taking  a  new  approach  to  reviewing 
proposed  concepts  that  will  provide  decision  makers  with  an  early 
opportunity  to  evaluate  trade-offs  among  alternative  approaches  to 
meeting  a  capability  need,  (2)  testing  portfolio  management  approaches  in 
selected  capability  areas  to  facilitate  more  strategic  choices  about  how  to 
allocate  resources  across  programs,  and  (3)  using  capital  budgeting  as  a 
potential  means  to  stabilize  program  funding.  While  promising,  these 
initiatives  do  not  fundamentally  change  DOD’s  existing  service-centric 
framework  for  making  weapon  system  investment  decisions. 


To  address  a  perceived  gap  between  DOD’s  major  decision-making 
processes  and  provide  a  department-level  means  to  assess  potential 
solutions  (materiel  and  non-materiel)  to  fill  a  validated  capability  need, 
DOD  is  testing  a  new  approach  to  a  Concept  Decision  review,  which  will 
take  place  after  a  warfighting  need  is  validated  by  the  JROC.  This  new 
approach  is  intended  to  focus  attention  on  the  affordability  and  feasibility 
of  potential  solutions  and  generate  early  cost,  schedule,  and  performance 
trade-offs  prior  to  the  point  of  a  significant  investment  commitment.  As 
currently  proposed,  the  Concept  Decision  will  be  informed  by  a  newly 
required  Evaluation  of  Alternatives  that  will  integrate  the  Functional 
Solutions  Analysis  conducted  under  JCIDS  with  the  Analysis  of 
Alternatives  conducted  under  the  acquisition  system  and  lay  out  the 
relative  merits  and  limitations  of  potential  solutions.  Furthermore,  concept 
decision  reviews  will  be  implemented  by  a  tri-chair  board  consisting  of 
lead  decision  makers  from  the  JCIDS,  PPBE,  and  DAS  processes.  While 
promising,  the  Concept  Decision  review  largely  reinstitutes  a  review  point 
that  already  existed  but  was  only  intermittingly  used.  For  Concept 
Decision  reviews  to  be  effective,  DOD  will  have  to  establish  enforcement 
and  accountability  mechanisms  to  ensure  the  reviews  are  actually 
implemented.  In  addition,  the  extent  to  which  the  concept  reviews  can 


34 Defense  Acquisition  Performance  Assessment  (January  2006);  Defense  Science  Board 
Summer  Study  on  Transformation:  A  Progress  Assessment  (February  2006);  Beyond 
Goldwater-Nichols:  U.S.  Government  and  Defense  Reform  for  a  New  Strategic  Era,  Phase 
2  Report  (July  2005);  and  Joint  Defense  Capabilities  Study  (January  2004). 
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achieve  desired  effects  will  depend  on  what  authority  Concept  Decisions 
carry  and  who  will  be  held  accountable,  particularly  in  light  of  the  service- 
dominated  investment  structure  that  currently  exists. 

The  department  has  also  begun  to  pilot-test  capability-based  portfolio 
management,  selecting  four  joint  capability  areas  to  focus  on — joint 
command  and  control,  joint  net-centric  operations,  battlespace  awareness, 
and  joint  logistics.  The  intent  is  to  enable  the  department  to  develop  and 
manage  capabilities,  as  opposed  to  simply  individual  programs,  and 
enhance  the  integration  and  interoperability  within  and  across  sets  of 
capabilities.  Each  portfolio  is  being  structured  somewhat  differently  to 
help  the  department  determine  how  best  to  proceed  with  portfolio 
management.  All,  however,  are  intended  to  focus  initially  on  existing 
programs  and  to  operate  within  DOD’s  existing  decision-making 
framework.  The  portfolios  are  largely  advisory  and  will,  as  a  first  step, 
provide  input  to  decisions  made  through  the  JCIDS,  PPBE,  and  DAS 
processes.  At  this  point,  the  capability  portfolio  managers  have  not  been 
given  direct  authority  to  manage  fiscal  resources  and  make  investment 
decisions.  Without  portfolios  in  which  managers  have  authority  and 
control  over  resources,  DOD  is  at  risk  of  continuing  to  develop  and 
acquire  systems  in  a  stovepiped  manner  and  of  not  knowing  whether  its 
systems  are  being  developed  within  available  resources. 

DOD  is  also  examining  the  use  of  capital  accounts  as  a  potential  means  of 
stabilizing  program  funding,  which  has  long  been  cited  as  a  significant 
issue  in  program  management.  This  capital  budgeting  pilot  initiative  is  in 
the  early  stages  of  planning,  and  the  specifics  of  how  such  accounts  will 
be  implemented  are  being  developed,  but  the  intent  is  for  DOD  to  commit 
a  set  amount  of  funding  for  the  development  portion  of  a  project  and  hold 
to  that  commitment  by  not  adjusting  funding  up  or  down  until  the  product 
is  delivered.  In  addition  to  resource  constraints,  programs  would  be  given 
a  fixed  amount  of  time  to  get  from  one  milestone  to  the  next.  If  successful, 
this  initiative  could  represent  a  step  toward  stabilizing  long-term  costs 
within  major  defense  acquisition  programs,  as  well  as  a  strengthening  of 
the  ability  of  program  managers  to  conduct  long-term  planning  and  control 
costs.  However,  for  this  initiative  to  be  effective,  DOD  will  need  to 
overcome  long-standing  problems  it  has  had  in  starting  programs  without 
sufficient  knowledge  of  the  costs,  requirements,  and  technologies  needed 
to  develop  proposed  weapon  systems.  Unless  this  changes,  it  is  unlikely 
that  capital  accounts  will  lead  to  increased  program  stability. 
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Conclusions 


While  DOD  has  increasingly  strengthened  its  ability  to  operate  as  a  joint 
force  on  the  battlefield,  the  department’s  organizational  structures, 
processes,  and  practices  for  planning  and  acquiring  weapon  systems  are 
not  similarly  joint.  Put  simply,  DOD  largely  continues  to  base  its 
investment  decisions  on  service-driven  analyses  that  do  not  provide  an 
enterprise-level  understanding  of  overall  warfighting  needs  and  on 
individual  platforms  rather  than  broader  sets  of  capabilities.  In  contrast, 
successful  commercial  companies  use  an  integrated  portfolio  management 
approach  to  focus  early  investment  decisions  on  products  collectively  at 
an  enterprise  level  and  to  ensure  there  is  a  sound  basis  to  justify  the 
commitment  of  resources.  By  following  a  disciplined,  integrated  process — 
where  the  relative  pros  and  cons  of  market  opportunities  and  competing 
product  proposals  are  assessed  based  on  available  resources  and 
customer  needs,  and  where  tough  decisions  about  which  investments  to 
pursue  are  made — companies  are  able  to  reduce  duplication  between 
business  units,  move  away  from  organizational  stovepipes,  and  effectively 
support  each  new  development  program  they  commit  to.  Until  DOD  takes 
a  joint,  portfolio  management  approach  to  weapon  system  acquisition — 
with  functionally  aligned  entities  that  have  the  requisite  responsibility, 
authority,  and  control  over  resources — it  will  continue  to  struggle  to 
effectively  prioritize  warfighting  needs,  make  informed  trade-offs,  and 
achieve  a  balanced  mix  of  weapon  systems  that  are  affordable,  feasible, 
and  provide  the  best  military  value  to  the  warfighter.  Committing  to  more 
programs  than  the  budget  can  support  and  approving  programs  based  on 
insufficient  knowledge  to  effectively  manage  risks  will  further  delay 
providing  critical  capabilities  to  the  warfighter  and  lead  to  lost 
opportunities  to  address  other  current  and  emerging  needs. 


Recommendations 


We  recommend  that  the  Secretary  of  Defense  implement  an  enterprise¬ 
wide  portfolio  management  approach  to  making  weapon  system 
investments  that  integrates  the  assessment  and  determination  of 
warfighting  needs  with  available  resources  and  cuts  across  the  services  by 
functional  or  capability  area.  To  ensure  the  success  of  such  an  approach, 
the  Secretary  should  establish  a  single  point  of  accountability  at  the 
department  level  with  the  authority,  responsibility,  and  tools  to  ensure 
that  portfolio  management  for  weapon  system  investments  is  effectively 
implemented  across  the  department. 

In  addition,  the  Secretary  should  ensure  that  the  following  commercial 
best  practices,  identified  in  this  report,  are  incorporated: 
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•  implement  a  review  process  in  which  needs  and  resources  are 
integrated  early  and  in  which  resources  are  committed 
incrementally  based  on  the  achievement  of  specific  levels  of 
knowledge  at  established  decision  points; 

•  prioritize  programs  based  on  the  relative  costs,  benefits,  and  risks 
of  each  investment  to  ensure  a  balanced  portfolio; 

•  require  increasingly  precise  cost,  schedule,  and  performance 
information  for  each  alternative  that  meets  specified  levels  of 
confidence  and  allowable  deviations  at  each  decision  point  leading 
up  to  the  start  of  product  development; 

•  establish  portfolio  managers  who  are  empowered  to  prioritize 
needs,  make  early  go/no-go  decisions  about  alternative  solutions, 
and  allocate  resources  within  fiscal  constraints;  and 

•  hold  officials  at  all  levels  accountable  for  achieving  and  maintaining 
a  balanced,  joint  portfolio  of  weapon  system  investments  that  meet 
the  needs  of  the  warfighter  within  resource  constraints. 

We  also  recommend  that  the  Secretary  take  steps  to  support  department- 
level  decision  makers  and  portfolio  managers  by  developing  a  stronger 
joint  analytical  capability  to  assess  and  prioritize  warfighting  needs. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  The 
comments  appear  in  appendix  II. 

DOD  concurred  with  the  majority  of  our  recommendations  and  partially 
concurred  with  two.  Generally,  in  responding  to  these  recommendations, 
DOD  stated  that  it  is  undertaking  several  initiatives  and  pilot  efforts  to 
improve  the  department’s  approach  to  investment  and  program  decision 
making,  and  that  implementation  of  any  new  business  rules  will  be 
contingent  upon  the  outcome  of  these  initiatives.  The  department  also 
stated  that  it  is  experimenting  with  portfolio  management,  related 
authorities  and  organizational  constructs,  and  integrated  decision-making 
processes. 

We  believe  that  these  initiatives  and  pilot  efforts  may  be  steps  in  the  right 
direction,  but  we  are  concerned  that  they  do  not  go  far  enough  to  address 
the  systemic  cultural  and  structural  problems  identified  in  this  report. 

DOD  has  attempted  many  similar  acquisition  reform  efforts  over  the  past  3 
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decades,  including  significant  revisions  to  both  defense  requirements  and 
acquisition  policy.  However,  despite  these  efforts,  weapon  system 
acquisition  programs  continued  to  experience  cost  overruns,  schedule 
slips,  and  performance  shortfalls.  The  department’s  current  initiatives  are 
likely  to  face  the  same  fate  because  they  do  not  fundamentally  change 
DOD’s  service-centric  framework  or  sufficiently  integrate  its  decision¬ 
making  processes  for  making  weapon  system  investments. 

DOD  did  not  provide  comments  regarding  our  recommendation  that  the 
Secretary  establish  a  single  point  of  accountability  at  the  department  level 
with  the  authority,  responsibility,  and  tools  to  ensure  that  portfolio 
management  for  weapon  system  investments  is  effectively  implemented 
across  the  department.  We  believe  that  a  single  point  of  accountability  is 
necessary  to  successfully  implement  a  portfolio  management  approach 
and  integrate  DOD’s  fragmented  decision-making  processes  under  one 
senior  official  who  is  accountable  for  weapon  system  investment 
outcomes.  We  further  believe  that  our  recommendations  would  better 
position  DOD  to  make  tough,  knowledge-based  choices  among  potential 
weapon  system  investments. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Secretaries  of  the  Air  Force,  Army,  and  Navy;  and  the  Director  of  the 
Office  of  Management  and  Budget.  We  will  provide  copies  to  others  on 
request.  This  report  will  also  be  available  at  no  charge  on  GAO’s  Web  site 
at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  call  me  at  (202)  512-4841  (sullivanm@gao.gov).  Key  contributors  to 
this  report  were  John  Oppenheim,  Assistant  Director;  Lily  Chin; 

John  Krump;  Matthew  Lea;  Travis  Masters;  Sean  Seales;  Karen  Sloan; 
Susan  Woodward;  and  Rebecca  Yunnan. 


Michael  J.  Sullivan 
Director,  Acquisition 

and  Sourcing  Management 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


This  report  examines  the  Department  of  Defense’s  (DOD)  requirements 
identification  and  resource  allocation  processes  for  major  weapons 
systems.  The  primary  focus  is  on  identifying  successful  private-sector 
principles  and  practices  that  could  be  adopted  by  DOD  to  help  improve 
stability  in  weapon  system  acquisition  programs.  Specifically,  our 
objectives  were  to  (1)  identify  best  practices  of  successful  commercial 
companies  for  ensuring  that  they  pursue  the  right  mix  of  programs  to  meet 
the  needs  of  their  customers  within  resource  constraints  and  (2)  compare 
DOD’s  enterprise-level  processes  for  investing  in  weapon  systems  to  those 
practices.  Our  work  was  conducted  between  March  2006  and  February 
2007,  in  accordance  with  generally  accepted  government  auditing 
standards. 

We  analyzed  the  outputs  of  DOD’s  investment  decision-making  support 
processes — the  requirements  determination  process  known  as  JCIDS  and 
the  resource  allocation  process  known  as  PPBE — using  criteria 
established  in  DOD  policy  and  in  previous  GAO  reports.  We  identified 
impacts  of  the  existing  processes  by  analyzing  quantitative  and  qualitative 
data  on  DOD  spending  trends,  conducting  interviews  with  DOD  officials, 
and  reviewing  previous  reports  by  GAO  and  by  other  knowledgeable  audit 
and  research  organizations.  In  addition,  we  met  with  officials  representing 
the  Office  of  the  Secretary  of  Defense,  Joint  Staff,  and  military  services.  At 
each  of  these  locations,  we  conducted  interviews  that  helped  us  describe 
the  current  condition  of  DOD’s  requirements  identification  and  resource 
allocation  processes.  We  also  reviewed  DOD  and  military  service  policies 
and  funding  documents  pertaining  to  the  DOD  requirements  identification 
process  and  resource  allocation  decisions  for  major  weapons  systems. 
Specifically,  we  reviewed  the  contents  of  14  unclassified  Initial  Capability 
Documents  that  were  finalized  after  June  24,  2003 — the  publication  date 
for  the  JCIDS  instruction — to  assess  the  extent  to  which  they  contained 
cost  and  technical  feasibility  information.  Those  14  ICDs  were 
unclassified,  weapon  system-related  ACAT  I,  II,  or  III  ICDs  that  were 
contained  in  the  Joint  Staff  requirements  database.  We  relied  on  previous 
GAO  reports  that  highlight  both  the  symptoms  and  causes  of  unstable 
requirements  and  funding  in  DOD  weapons  acquisition  programs.  A  list  of 
these  reports  can  be  found  at  the  end  of  this  report.  In  addition,  we 
reviewed  recent  key  studies  and  reports  addressing  acquisition  reform 
issues  by  the  Center  for  Strategic  International  Studies,  Institute  for 
Defense  Analysis,  the  U.S.  Naval  War  College,  the  Defense  Acquisition 
Performance  Assessment  Project,  the  Joint  Defense  Capabilities  Study 
Team,  the  Joint  C4ISR  Decision  Support  Center,  the  Defense  Science 
Board,  and  the  2001  and  2005  Quadrennial  Defense  Reviews. 
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We  also  reviewed  pertinent  literature  from  authoritative  corporate, 
academic,  and  professional  organizations,  to  identify  commercial  best 
practices  and  processes  that  could  be  used  by  DOD  to  improve  its  weapon 
system  investment  decision-making  processes.  In  addition  we  conducted 
case  studies  of  five  leading  commercial  companies.  In  selecting  them,  we 
sought  to  identify  companies  that  were  recognized  in  the  literature  for  best 
practices,  had  large  and  diversified  portfolios  of  products,  and  make 
significant  investments  in  the  development  and  production  of  new 
products.  For  each  of  the  companies,  we  interviewed  management 
officials  knowledgeable  about  their  requirements  identification  and 
resource  allocation  activities,  to  gather  consistent  information  about 
processes,  practices,  and  metrics  the  companies  use  to  help  achieve 
successful  product  development  outcomes.  Below  are  descriptions  of  the 
five  companies  featured  in  this  report: 

Motorola 

Motorola  is  a  Fortune  100  global  communications  leader  that  provides 
seamless  mobility  products  and  solutions  across  broadband,  embedded 
systems,  and  wireless  networks.  According  to  Motorola’s  2005  Corporate 
Profile,  the  company  is  the  market  leader  in  mission  critical  wireless 
communication  systems,  two-way  radios,  embedded  telematics  systems, 
digital  set-top  shipments,  cable  modem  shipments,  digital  head-ends, 
embedded  computer  systems  for  communication  applications,  CDMA 
infrastructure  sales  (excluding  the  United  States),  and  second  in  world 
wide  wireless  handsets.  Motorola  achieved  net  sales  of  $31,323  billion  and 
spent  $3,060  billion  on  research  and  development  in  2004.  The  corporation 
has  approximately  68,000  employees,  in  320  facilities,  spanning 
73  countries.  We  met  with  the  management  of  Motorola’s  Government  & 
Enterprise  Mobility  Solutions  and  Global  Telecom  Solutions  sectors  in 
Schaumburg,  Illinois. 

International  Business  Machines  (IBM) 

IBM  is  one  of  the  world’s  largest  technological  companies,  spending  about 
$3  billion  annually  on  research  and  development  activities.  It  is  the  largest 
supplier  of  hardware,  software,  and  information  technology  services.  With 
3,248  U.S.  patents,  IBM  earned  more  patents  than  any  other  company  for 
the  12th  consecutive  year  in  2004.  In  the  past  4  years,  IBM  inventors 
received  more  than  13,000  patents — approximately  5,400  more  than  any 
other  patent  recipient.  IBM  has  over  329,000  employees  worldwide.  We 
met  with  managers  from  IBM  Integrated  Product  Development  (IPD)  in 
Somers,  New  York. 
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Procter  &  Gamble  (P&G) 

Procter  &  Gamble  Corp.  (P&G)  is  a  leading  producer  of  consumer  goods. 

It  currently  leads  in  global  sales  and  marketshare  among  all  fabric  care, 
baby  care,  feminine  care,  and  hair  care  products.  It  currently  has  over 
130,000  employees  in  80+  countries.  Twenty-two  of  its  brands  have  annual 
gross  sales  exceeding  $1  billion  each.  In  fiscal  year  2005/2006,  P&G 
invested  $2,075  billion  or  3  percent  of  net  sales  in  research  and 
development  (R&D).  This  ranks  them  as  one  of  the  top  20  largest  research 

6  development  investors  among  U.S.-based  companies.  P&G  has  more 
Ph.D.s  working  in  labs  around  the  world  than  the  combined  science  and 
engineering  faculties  of  Harvard,  MIT,  and  Berkeley.  We  met  with  the 
management  of  P&G’s  New  Initiative  Delivery  team  in  Cincinnati,  Ohio. 

Eli  Lilly  Corporation 

Eli  Lilly  is  a  global  pharmaceutical  company  and  one  of  the  world’s  largest 
corporations.  It  was  founded  over  130  years  ago  and  currently  employs 
approximately  42,000  people  worldwide,  including  13,991  employed  at  its 
headquarters  in  Indianapolis,  Ind.  Approximately  8,336  employees 
(19  percent  of  the  total  work  force)  are  engaged  in  research  and 
development  (R&D);  clinical  research  is  conducted  in  over  50  countries; 
there  are  R&D  facilities  in  9  countries;  and  manufacturing  plants  in 
13  countries.  Its  products  are  marketed  in  143  countries.  Lilly’s  net  sales  in 
2005  were  $14.6  billion.  Eli  Lilly  strives  to  grow  sales  by  6  percent  to 

7  percent  each  year.  In  2005,  $3  billion  was  spent  on  R&D,  a  $334.4  million 
increase  from  the  previous  year.  Currently,  R&D  represents  20.7  percent  of 
sales.  Lilly’s  total  R&D  investment  in  the  last  5  years  from  continuing 
operations  was  $12.5  billion.  We  met  with  managers  from  Eli  Lilly’s 
Corporate  Headquarters  in  Indianapolis,  Ind. 

Caterpillar  Corporation 

Caterpillar  is  a  technology  leader  and  the  world’s  leading  manufacturer  of 
construction  and  mining  equipment,  diesel  and  natural  gas  engines,  and 
industrial  gas  turbines.  In  2005,  its  total  sales  and  revenues  were 
$36.3  billion,  and  its  total  R&D  expenditures  exceeded  $1  billion, 
compared  with  $20.5  billion  sales  and  $696  million  R&D  in  2001.  Between 
2001  and  2005,  the  average  return  on  equity  of  its  stockholders’  shares 
more  than  doubled.  Caterpillar  has  over  85,000  employees,  and  over 
105,000  people  are  employed  by  Caterpillar’s  dealers  worldwide.  We  met 
with  managers  responsible  for  Caterpillar’s  New  Product  Introduction 
(NPI)  process  in  Peoria,  Illinois. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION. 

TECHNOLOGY 
AND  LOGISTICS 

Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.  S.  Government  Accountability  Office 
441  G.  Street,  N.W. 

Washington,  DC  20548 


MAR  2  1  2007 


Dear  Mr.  Sullivan: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report, 
GAO-07-388  “BEST  PRACTICES:  An  Integrated  Portfolio  Management  Approach  to 
Weapon  System  Investments  Could  Improve  DoD’s  Acquisition  Outcomes,”  dated 
February  14, 2007  (GAO  Code  120516). 

The  DoD  concurs  with  five  recommendations  and  partially  concurs  with  two  of 
the  draft  report  recommendations.  The  rationale  for  our  position  is  included  in  the 
enclosure. 

We  appreciate  the  opportunity  to  comment  on  the  draft  report.  My  point  of 
contact  for  this  effort  is  Mr.  David  Crim,  (703)  697-5385,  david.crim@osd.mil 


Portfolio  Systems  Acquisition 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  DATED  FEBRUARY  14,  2007 
GAO-07-388  (GAO  CODE  120516) 

"BEST  PRACTICES:  AN  INTEGRATED  PORTFOLIO 
MANAGEMENT  APPROACH  TO  WEAPON  SYSTEM 
INVESTMENTS  COULD  IMPROVE  DOD’S  ACQUISITION 
OUTCOMES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense 
implement  an  enterprise-wide  portfolio  management  approach  to  making  weapon 
system  investments  that  integrates  the  assessment  and  determination  of  warfighting 
needs  with  available  resources  and  cuts  across  the  services  by  functional  or  capability 
area.  (p.  25/GAO  Draft  Report) 

POD  RESPONSE:  Concur.  As  noted  in  the  report,  in  response  to  the  2006 
Quadrennial  Defense  Review  (QDR)  and  other  recent  acquisition  reform  studies,  the 
DoD  is  undertaking  several  initiatives  and  pilot  activities  to  improve  the  department’s 
approach  to  investment  and  program  decision  making.  These  initiatives  take  into 
account  many  of  the  best  practices  employed  by  commercial  industry  as  described  in  the 
report,  including  corporate-level  priorities,  a  framework  linking  strategic  goals  to  plans 
and  budgets,  and  performance  metrics.  Implementation  of  any  new  business  rules  will 
be  contingent  upon  the  outcome  of  these  pilot  initiatives,  a  thorough  understanding  of 
the  best  overall  approach,  and  the  overall  benefits  gained. 

RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense 
implement  a  review  process  in  which  needs  and  resources  are  integrated  early  and  in 
which  resources  are  committed  incrementally  based  on  the  achievement  of  specific 
levels  of  knowledge  at  decision  points,  (p.  26/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  As  noted  in  the  report,  in  response  to  the  2006  QDR  and 
other  recent  acquisition  reform  studies,  the  DoD  is  undertaking  several  initiatives  and 
pilot  activities  to  integrate  the  department’s  major  decision  processes.  Concept 
Decision  and  Capital  Accounts  are  two  examples  of  such  initiatives  that  are  consistent 
with  this  report  recommendation.  Foundational  to  these  and  other  pilot  activities  is 
available  knowledge  to  support  senior  leader  decision  making  at  key  points  in  the  life 
cycle.  The  Department’s  efforts  to  promote  transparent  data,  along  with  its  major 
decision  processes  support  the  incremental  commitment  of  resources.  Implementation 
of  any  new  business  rules  will  be  contingent  upon  the  outcome  of  these  initiatives,  a 
thorough  understanding  of  the  best  overall  approach  and  the  overall  benefits  gained. 
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RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense 
prioritize  programs  based  on  the  relative  costs,  benefits,  and  risks  of  each  investment  to 
ensure  a  balanced  portfolio,  (p.  26/GAO  Draft  Report) 

POD  RESPONSE:  Partially  Concur.  As  noted  in  the  report,  in  response  to  the  2006 
QDR  and  other  recent  acquisition  reform  studies,  the  DoD  is  undertaking  several 
initiatives  to  enable  strategic  choice.  This  involves  balancing  resources  at  three  levels: 
strategic,  portfolio  and  program  levels.  Implementation  of  any  new  business  rules  will 
be  contingent  upon  the  outcome  of  initiatives  focused  in  this  area,  a  thorough 
understanding  of  the  best  overall  approach  and  the  overall  benefits  gained. 

RECOMMENDATION  4:  The  GAO  recommended  that  the  Secretary  of  Defense 
require  increasingly  precise  cost,  schedule,  and  performance  information  for  each 
alternative  that  meet  specified  levels  of  confidence  and  allowable  deviations  at  each 
decision  point  leading  up  to  the  start  of  product  development,  (p.  26/GAO  Draft 
Report) 

DOD  RESPONSE:  Concur.  As  noted  in  the  report,  in  response  to  the  2006  QDR  and 
other  recent  acquisition  reform  studies,  the  DoD  is  undertaking  several  initiatives  and 
pilot  activities  to  improve  the  department’s  approach  to  investment  and  program 
decision  making.  Implementation  of  any  new  business  rules  will  be  contingent  upon  a 
thorough  understanding  of  the  best  overall  approach  and  the  overall  benefits  gained. 

RECOMMENDATION  5:  The  GAO  recommended  that  the  Secretary  of  Defense 
establish  portfolio  managers  who  are  empowered  to  prioritize  needs,  make  early 
go/no-go  decisions  about  alternative  solutions,  and  allocate  resources  within  fiscal 
constraints,  (p.  26/GAO  Draft  Report) 

DOD  RESPONSE:  Partially  Concur.  As  noted  in  the  report,  in  response  to  the  2006 
QDR  and  other  recent  acquisition  reform  studies,  the  DoD  is  undertaking  several 
initiatives  and  pilot  activities  to  experiment  with  Capability  Portfolio  Management.  We 
have  selected  four  highly  joint  areas  with  which  to  initiate  a  portfolio  construct,  and  are 
experimenting  with  their  participation  and  integration  in  the  Department’s  major 
processes,  as  well  as  with  their  authorities  and  organizational  constructs. 

Implementation  of  any  new  business  rules  will  be  informed  by  the  experience  gained 
and  information  gathered  from  the  Capability  Portfolio  management  effort  and  based  on 
a  thorough  understanding  of  the  best  overall  approach  and  the  overall  benefits  gained. 

RECOMMENDATION  6:  The  GAO  recommended  that  the  Secretary  of  Defense  hold 
officials  at  all  levels  accountable  for  achieving  and  maintaining  a  balanced,  joint 
portfolio  of  weapon  system  investments  that  meet  the  needs  of  the  warfighter  within 
resource  constraints,  (p.  26/GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  In  response  to  the  2006  QDR,  the  Department  developed 
an  execution  roadmap  for  Institutional  Reform  and  Governance.  This  roadmap  is  aimed 
at  improving  the  defense  enterprise  by  creating  more  integrated  and  responsive  decision 
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making  processes,  organizations  and  business  practices.  The  roadmap  seeks  to  develop 
a  decision  management  approach  that  enables  a  clear  and  transparent  link  from  strategy 
to  outcomes.  The  approach  will  clearly  delineate  decision  making  responsibilities  of 
the  governance,  management  and  execution  levels  of  the  Department.  It  will  also 
enable  senior  leadership  to  focus  on  strategic  choice  and  empower  management  to  carry 
out  their  responsibilities  in  a  manner  that  ensures  transparency,  accountability,  and 
sound  performance  management.  An  example  acquisition  transformation  initiative  is 
the  utilization  of  a  senior-level  Tri-Chaired  Committee  that  integrates  Department-wide 
acquisition  processes  to  conduct  Concept  Decision  Reviews.  The  Concept  Decision 
Review  goal  is  to  ensure,  as  early  as  possible,  that  DoD  is  making  the  right  corporate 
investment  choices,  balancing  operational  and  programmatic  risks,  to  ensure  that  those 
choices  are  affordable,  and  that  any  resulting  non-materiel  solutions  and/or  materiel 
acquisitions  are  designed  for  lifecycle  success. 

RECOMMENDATION  7:  The  GAO  recommended  that  the  DoD  take  steps  to  support 
department  level  decisions-makers  and  portfolio  managers  by  developing  a  stronger 
joint  analytical  capability  to  assess  and  prioritize  warfighting  needs,  (p.  26/GAO  Draft 
Report) 

DOD  RESPONSE:  Concur.  As  part  of  the  overall  Institutional  Reform  and 
Governance  effort  described  above,  we  will  work  to  improve  the  Department’s  analytic 
framework,  build  more  transparent  business  information  across  the  Department, 
integrate  decision  processes  to  enable  strategic  choice  and  align  roles  and 
responsibilities  in  a  way  that  maximizes  decision  making  effectiveness  across  the 
enterprise.  As  an  example,  the  Department  is  continuing  to  develop  its  weapon  system 
readiness  and  sustainment  modeling  capabilities.  Such  models  are  aimed  at  enhancing 
the  ability  of  acquisition  and  sustainment  professionals  to  assess,  for  example,  trade¬ 
offs  between  alternative  investments  in  elements  such  as  sustainment  capabilities, 
support  structures,  cycle  times,  reliability,  and  alternative  policies.  The  goal  of  these 
modeling  capabilities  is  to  ensure  that  DoD  can  make  the  intelligent,  informed  resource 
application  decisions  that  will  optimize  acquisition  and  sustainment  operations  and 
maximize  achieved  materiel  readiness  at  optimum  total  weapon  systems  ownership  cost. 
Such  modeling  capabilities  support  the  ongoing  integration  of  acquisition  and 
sustainment  activities  into  an  end-to-end  continuum. 
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Defense  Acquisitions:  Stronger  Management  Practices  Are  Needed  to 
Improve  DOD’s  Software-Intensive  Weapon  Acquisitions.  GAO-04-393. 
Washington,  D.C.:  March  1,  2004. 

Best  Practices:  Setting  Requirements  Differently  Could  Reduce  Weapon 
Systems’  Total  Ownership  Costs.  GAO-03-57.  Washington,  D.C.:  February 
11,  2003. 

Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early 
Improves  Acquisition  Outcomes.  GAO-02-701.  Washington,  D.C.:  July  15, 
2002. 

Defense  Acquisitions:  DOD  Faces  Challenges  in  Implementing  Best 
Practices.  GAO-02-469T.  Washington,  D.C.:  February  27,  2002. 

Best  Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to 
Better  Weapon  System  Outcomes.  GAO-Ol-288.  Washington,  D.C.:  March  8, 
2001. 

Best  Practices:  A  More  Constructive  Test  Approach  Is  Key  to  Better 
Weapon  System  Outcomes.  GAO/NSIAD-OO-199.  Washington,  D.C.:  July  31, 
2000. 

Defense  Acquisition:  Employing  Best  Practices  Can  Shape  Better 
Weapon  System  Decisions.  GAO/T-NSIAD-OO-137.  Washington,  D.C.:  April 
26,2000.  ~ 

Best  Practices:  DOD  Training  Can  Do  More  to  Help  Weapon  System 
Programs  Implement  Best  Practices.  GAO/NSLAD-99-206.  Washington, 
D.C.:  August  16,  1999. 
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Related  GAO  Products 


Best  Practices:  Better  Management  of  Technology  Development  Can 
Improve  Weapon  System  Outcomes.  GAO/NSIAD-99-162.  Washington, 
D.C.:  July  30,  1999. 

Defense  Acquisitions:  Best  Commercial  Practices  Can  Improve  Program 
Outcomes.  GAO/T-NSLAD-99-116.  Washington,  D.C.:  March  17,  1999. 

Defense  Acquisition:  Improved  Program  Outcomes  Are  Possible. 
GAO/T-NSIAD-98-123.  Washington,  D.C.:  March  17,  1998. 

Best  Practices:  DOD  Can  Help  Suppliers  Contribute  More  to  Weapon 
System  Programs.  GAO/NSIAD-98-87.  Washington,  D.C.:  March  17,  1998. 

Best  Practices:  Successful  Application  to  Weapon  Acquisition  Requires 
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February  24,  1998. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “Subscribe  to  Updates.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Gloria  Jarmon,  Managing  Director,  JarmonG@gao.gov  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  D.C.  20548 

Public  Affairs 


Paul  Anderson,  Managing  Director,  AndersonPl@gao.gov  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  D.C.  20548 
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